Hand-Eye / Robot-World Calibration
RANSAC Problem
VisualGeometryToolkitEstimation.HandEyeProblem — Type
HandEyeProblem{S} <: AbstractRansacProblem{2}RANSAC problem for rigid hand-eye / robot-world calibration. Solves A_ij X = X B_ij for the constant X = T_RP ∈ SE(3) from a stream of relative motion pairs. Minimal sample is 2 motion pairs.
Type parameters:
S <: AbstractSampler: sampling strategy
Constructor
# From synchronized absolute-pose observations, one A_i => B_i per frame
cs = A .=> B # A_i = T_WR (mocap), B_i = T_CP (PnP), in time order
problem = HandEyeProblem(cs)Fields
motions::Vector{Tuple{Rigid3, Rigid3}}: (Aij, Bij) relative-motion pairsaxes::Vector{SVector{3, Float64}}: the rotation vectorlog_so3(R_A)of each rig motion, cached fortest_sample's parallel-axis testchar_length::Float64: characteristic length used to balance the rotation and translation residual blocks (set automatically from data, override via the constructor)min_rel_angle::Float64: the smallest rotation (rad) every motion has — checked once, here_sampler::S: minimal-sample sampler
The inner constructor HandEyeProblem{S}(motions, char_length, min_rel_angle, sampler) is THE rotation gate: it throws an ArgumentError when a motion's rig or pattern rotation is below min_rel_angle (throws rather than drops, so the sampler built for motions stays consistent). The keyword constructor filters its relative motions by the same predicate first, so it never throws there; a problem built from raw motions cannot bypass the gate. A near-identity motion's axis is noise-dominated, and its hypotheses scatter far outside the first-order covariance they are scored with: the predictive NEES tail (228/6 at 1 mrad, median ratio still 1.03) is exactly these motions, and admitting only motions of ≥ 5° calibrates it (Monte Carlo, ransac_mc_calibration (b)).
VisualGeometryToolkitEstimation.Rigid3 — Type
Rigid3The unlabeled rigid motion of hand-eye calibration: a concrete EuclideanMap{3, Float64} with a RotMatrix{3} rotation and a Vec3 translation between two UnlabeledFrame{3}s. It is the model type of HandEyeProblem, the currency of its relative motions (A_ij, B_ij), and the estimate of find_hand_eye on plain poses. Its translation is a plain number in the poses' own length unit; for physical poses (translations in lengths) find_hand_eye returns the same map with its translation in metres, not a Rigid3.
Not a camera extrinsics: the motion algebra inv(A[j]) ∘ A[i] composes maps whose frames a labeled Extrinsics3 would reject.
Fitting Pipeline
VisualGeometryToolkitEstimation.find_hand_eye — Function
find_hand_eye(cs; scoring,
max_trials, min_trials,
local_optimization_strategy,
motion_pairs=:consecutive,
char_length=nothing, min_rel_angle,
config=nothing) -> RansacEstimateRobust hand-eye / robot-world calibration via RANSAC + IRLS. Every default is configs/default.toml's ([ransac], [pipelines.hand_eye]); a keyword overrides.
Arguments
cs— oneA_i => B_icorrespondence per frame, in time order: the rig-in-world pose (from motion capture) and the pattern-in-camera pose (from PnP; labeledExtrinsics3accepted — unlabeled at the problem boundary).
Keywords
scoring— the RANSAC scorer, its width in the TRANSLATION's length unit (the scene's, as the poses carry it): the residual stacks the translation error with the rotation error scaled by the characteristic length,L·θ(seeHandEyeProblemfor the unit balance). A scorer you supply carries that length (MsacScoring(HandEyeProblem; threshold = 0.005)). When omitted it is the class's, unit-free:configs/default.toml[pipelines.hand_eye] scoring(MSAC,threshold = 0.01) states a FRACTION of the problem's characteristic lengthL, so the default threshold is0.01·L— the same whether the poses are in metres or millimetres (a rotation error of 0.01 rad ≈ 0.57° reaches it).motion_pairs::Symbol = :consecutive—:consecutiveusesN-1consecutive-frame relative motions;:alluses every pair(i, j)withi < j.char_length— characteristic length forwarded toHandEyeProblem(default: median rig translation per motion).min_rel_angle— the smallest rotation (rad) a relative motion may have; motions below it are dropped at construction (default[robot_world_hand_eye]min_rel_angle_deg— seeHandEyeProblem).max_trials,min_trials— RANSAC trial bounds.local_optimization_strategy— default[pipelines.hand_eye] local_optimization_strategy— HandEyeProblem has no chart lift; its weighted refit (Levenberg–Marquardt on the so(3)⊕R³ chart ofRigid3) is selected byrefine!(lo_ws, ::HandEyeProblem, θ::Rigid3, weights).config— overrideRansacConfig; the other keywords asfind_homography.
Units
The poses are plain or physical, all one form. Physical poses (translations in lengths, e.g. EuclideanMap(R, t .* m)) are read in metres at entry — exactly, the stored value — the search runs on plain metres, and X comes back physical, its translation in metres, equal after ustrip(m, ·) to the plain search's on the poses' ustrip(m, ·) with the same rng. A supplied char_length is then a length too (5mm; a plain number beside physical poses is a DimensionError); a supplied scorer's width is a plain number in METRES (a scorer holds no unit). The residuals, rms and the scale (noise_scale(result)) are plain numbers in the translation's unit: metres for physical poses. Plain poses give a plain X in their own unit, as before.
Returns an FitResult{Rigid3, RansacMetadata} whose estimate is X = T_RP (for physical poses, X is the same map with its translation in metres). The nuisance camera-in-world transform Y is not computed by this pipeline. iterations(result) is the number of trials (hypotheses drawn); test found_model(result) before reading the model (see RansacEstimate).
Examples
A = [Rigid3(...) for _ in 1:N] # mocap
B = [Rigid3(...) for _ in 1:N] # PnP
result = find_hand_eye(A .=> B)
X = estimate(result) # T_RP, the calibrationVisualGeometryToolkitEstimation.refine_hand_eye — Function
refine_hand_eye(X, cs; motion_pairs = :consecutive, char_length = nothing, min_rel_angle,
optimization_strategy, scale | width, workspace = nothing)
-> FitResult{M, RefineMetadata}The hand-eye transform X = T_RP refined against the A_i => B_i pose correspondences cs from X itself, on find_hand_eye's HandEyeProblem (motion_pairs, char_length and min_rel_angle build it, as there; its 6-residual, codimension 6, in the poses' length unit with rotation entering as char_length·angle): the keywords, the result and the refusals are those of every refine_*, documented on refine_homography (default strategy: [refine.hand_eye] optimization_strategy). Fewer than 2 motions: the guess back unrefined, record (ReturnCode.InitialFailure, NoRule) (retcode_symbol:degenerate). A scale or width is in the poses' length unit.
Units
As find_hand_eye: plain or physical poses, all one form (X too); physical ones are read in metres and X comes back in metres, a supplied char_length then a length and a scale or width a plain number in metres.
Result Type
find_hand_eye returns a FitResult{Rigid3, RansacMetadata} (RansacEstimate{Rigid3}, with Rigid3 an unlabeled EuclideanMap{3, Float64}). The .estimate field is the recovered rigid transform X = T_RP (pattern in rig). The nuisance camera-in-world transform Y = T_WC is not returned — it drops out of the AX = XB formulation. The remaining diagnostics (inlier_mask(result), .residuals, weights(result), .iterations — for RANSAC the number of trials, hypotheses drawn — .converged, …) match the standard RansacEstimate surface.
Formulation
The pipeline implements the classical robot-world / hand-eye problem, eliminated via the relative-motion ratio:
(A_j⁻¹ A_i) · X = X · (B_j⁻¹ B_i) ⇔ A_ij · X = X · B_ijA_i = T_WR (rig in world, e.g. from motion capture) and B_i = T_CP (pattern in camera, e.g. from PnP). With N observations, motion_pairs = :consecutive (default) builds N - 1 relative motions from adjacent frames; :all uses every (i, j) with i < j.
The minimal sample is 2 motion pairs (Tsai–Lenz solvability condition: two non-parallel rotation axes). The minimal solver is the closed-form Tsai–Lenz algorithm (modified-Rodrigues LLS for rotation, then a 3×3 linear system for translation). The per-pair residual is a decoupled 6-vector
g_i(X) = [ L · Log(R_A R_X R_B^T R_X^T) ; R_A t_X + t_A − R_X t_B − t_X ] ∈ ℝ⁶where L = char_length (default: median rig-translation per motion) balances rotation (radians) and translation (length) units.
Local Optimization
Hand-eye uses Irls{TLSShape}() LO out of the box (least squares on the inliers at the default MSAC scorer's threshold, 0.01·L). The per-iteration M-step for HandEyeProblem is a Levenberg–Marquardt refit on the 6-DOF decoupled chart about the current X — R_X = exp(δω)·R₀ plus an additive update t_X = t₀ + δt (VisualGeometryToolkitCore's RigidExpManifold6, the chart of constraint_with_jacobians' Gθ and of the solver covariance) — minimizing Σᵢ wᵢ ‖gᵢ‖² over the stacked 6-vectors. It runs VisualGeometryToolkitCore's chart LM (NLLSsolver, through lmsolve_chart!) with a ForwardDiff Jacobian of the explicit residual: that Gθ is only the near-optimum approximation of ∂g/∂δ, so the refit does not use it. Each RANSAC hypothesis is Tsai–Lenz fitted on its own two motions by the same refit (fit_hypothesis), the estimator the solver covariance describes.
Identifiability
The problem requires at least two relative motions with non-parallel rotation axes (Tsai & Lenz 1989; Chen 1991). With a smooth trajectory of N ≫ 2 poses, this is satisfied generically. A degenerate dataset (e.g. the rig only rotated about a single axis throughout the recording) leaves the component of X about that axis unobservable.
HandEyeProblem admits only relative motions that rotate at least min_rel_angle (configs/default.toml[robot_world_hand_eye]min_rel_angle_deg — the same authority fit_robot_world_hand_eye's init reads): its constructor is the one gate, so every motion a sample draws has succeeded at it. test_sample then rejects a minimal sample whose two axes lie within ~2.5° of each other. Both ill-condition X, and near-identity motions put a tail on the hypotheses' scatter that their first-order covariance does not predict. If you suspect trajectory-level degeneracy across the full dataset, inspect the rotation axes of the relative motions A_ij directly before relying on the result.
Limitations
- No reprojection refinement here: because
Y = T_WCis eliminated, this pipeline cannot refine on pixel reprojection error (that requiresYto project pattern points into pixels). The robot-world form below keepsY:fit_robot_world_hand_eyethenrefine_robot_world_hand_eyeminimizesΣ ‖uᵢⱼ − project(camera, Y⁻¹ ∘ Aᵢ ∘ Xₘ (Pⱼ))‖²overYand everyXₘ. - Single global characteristic length: the default
Lis a scalar set from the data; the rotation-vs-translation trade-off is not per-frame heteroscedastic. See Loss Variants for the design alternatives (two-scale Mahalanobis, per-pair noise model, iter-profiledL) — none are currently implemented.
Robot-World Calibration (Y kept)
A_i X_m = Y B_i^m with one shared Y = T_WC and per-marker X_m = T_RPm: fit_robot_world_hand_eye is the staged linear solve from per-(frame, marker) poses (its covariance is the first-order covariance of the staged estimator itself, propagated through each stage's stationarity conditions by the implicit function theorem, and Inf along unobservable directions); refine_robot_world_hand_eye refines it against the markers' features — the reprojection error, camera and rig poses fixed — on M + 1RigidExpManifold6 charts with NLLSsolver's Levenberg–Marquardt (one pose block per marker and one residual block per point, so any number of markers with nothing compiled per count) — under the refine contract's strategy (Triggs{WelschShape}() by default, at the caller's scale or width; optimization_strategy = nothing for least squares) — and reports the covariance of that estimator's own optimum, ŝ²·implicit_covariance, which the Monte Carlo gate holds calibrated (Y, each X_m, the joint vector, held-out reprojection, frame count, field of view, marker count and a poor seed).
The staged fit takes the robust-estimation contract's fixed-scale IRLS: optimization_strategy = Irls{K}() with σ_rot, σ_trans (the whitening) and exactly one of scale or width on the whitened 6-residual. Each stage is reweighted (Kabsch pairs, chordal means, the translation least squares), so a handful of gross poses — sparse frames a registration placed tens of degrees wrong — get weight ≈ 0 instead of tilting Y and every X_m.
VisualGeometryToolkitEstimation.fit_robot_world_hand_eye — Function
fit_robot_world_hand_eye(markers; σ_rot=nothing, σ_trans=nothing, min_rel_angle,
optimization_strategy = nothing, scale | width, tol)
-> FitResult{RobotWorldHandEyeCalibration{K}, RobotWorldHandEyeMetadata{K, Float64}}min_rel_angle defaults to configs/default.toml[robot_world_hand_eye]min_rel_angle_deg (5°), converted to radians.
Staged linear robot-world / multi-marker hand-eye calibration A_i X_m = Y B_i^m: one shared Y = T_WC and per-marker X_m = T_RPm, fit from per-(frame, marker) pose observations. No RANSAC: with a pose prior (mocap) contamination is pre-gateable at the call site, or the fit downweights it itself with a fixed-scale IRLS (optimization_strategy, below).
Robot-world/hand-eye calibration is the literature's name for AX = YB (Zhuang, Roth & Shah 1994; Dornaika & Horaud 1998). Ours is the EYE-TO-HAND configuration — the camera fixed in the world, the targets on the moving rig — with frames in the stack's T_AB notation: Y = T_WC, X_m = T_RPm, A_i = T_WR (the rig pose at frame i) and B_i^m = T_CPm (marker m in the camera at frame i). Function names do not encode frames.
The estimate is the staged linear solve: Kabsch on pooled relative-motion axes (R_Y), chordal means per marker (R_Xm), then one joint linear least-squares solve for ALL translations — exactly optimal in the translations given the rotations. A pose-residual NLLS refinement was deliberately removed: on real single-distance calibration data it absorbed an unmodeled systematic along a weakly constrained direction and extrapolated worse than this solution (see the documentation's "Robot–World Hand–Eye Calibration" page). Observation-level (reprojection) refinement is the sound upgrade path.
Arguments
markers::AbstractDict{K, <:AbstractVector{<:Correspondence}}— per marker id, the correspondences between rigid transformsA_i => B_i^mof every frame it is seen in: the rig poseT_WR(e.g. mocap) and the marker-in-camera poseT_CPm(e.g. PnP); any frame labels accepted,unlabeled at the boundary. The key is the marker's identity (any type withisless); the result'sidsare the keys sorted, andXis keyed the same. A frame seen by several markers repeats itsA_iin each marker's vector — nothing is associated by index.
Keywords
σ_rot,σ_trans— per-block noise scales (rad, translation units) balancing the diagnostic 6-vector residual[Log(R̂ᵀR)/σ_rot; (t−t̂)/σ_trans]. Whennothing, profiled as the RMS of the corresponding residual block at the estimate (equalizes block contributions).min_rel_angle— a within-marker relative motion enters the Kabsch stage only when both its rig and its marker motion rotate at least this angle (near-identity motions are uninformative) — the same predicate asfind_hand_eye's.optimization_strategy—nothing(the default: the staged linear solve above, least squares), or anIrls{K}strategy: the staged solve reweighted per observation at ONE loss width on the whitened 6-vector residual (codimension 6), iterated totolor the strategy'smax_iter. Each stage takes the weights: a Kabsch pair weighswₖ₁·wₖ₂, the chordal means and the translation least squareswₖ. A gross observation (a pose a registration placed tens of degrees and metres wrong) then gets weight ≈ 0 instead of tiltingYand everyX. The robust fit REQUIRESσ_rotandσ_trans: they are the whitening, the units of its scale, and profiling them from the contaminated residuals would re-estimate a scale the fit holds fixed (ArgumentErrorwithout them). A Triggs strategy is refused (a refine's:refine_robot_world_hand_eye).scale|width— with anIrlsstrategy, EXACTLY ONE of:scale = σ(the whitened residual's noise scale per component, dimensionless: 1 whenσ_rot/σ_transare the true noise),scale = result(an upstream robust result'snoise_scale), orwidth = w(the loss width on the whitened residual norm). The width isw, orc · σwithcthe strategy's efficiency constant, by default the shape's at codimension 6 (efficiency_constant(K, 6): Huber 1.8693, Welsch 3.9646). Neither, both, or one given to the least-squares fit is anArgumentError.tol— the robust fit's convergence: an iteration moves every transform by less thantolin the whitened units (rotation angle /σ_rot, translation /σ_trans); default configs/default.toml[robot_world_hand_eye] tol.
The covariance is that of the staged estimate itself, first order in the observation noise ŝ²·diag(σ_rot², σ_trans²) of the pose residual above (ŝ² = RSS/dof): the rotation stages optimize different functionals than that residual, so the estimate is not its stationary point and the residual's Gauss–Newton ŝ²(JᵀJ)⁻¹ would understate the scatter; instead the implicit function theorem on the stages' own stationarity conditions (Kabsch, chordal means, translation normal equations) carries the noise through. It is reported with the observability diagnostics (min_eigval/min_eigvec of the residual's JᵀJ), and is Inf when JᵀJ is rank-deficient. A robust fit reports the same at its final weights, held fixed: the weighted stage conditions, √w-scaled residual rows, ŝ² the weighted RSS over the weighted dof; rms is weighted and n the support (observations of positive weight). The metadata's weights (one per observation, marker by marker in ids order), scale, scale_source and loss_width are the robust-estimation contract's record; least squares carries unit weights and ScaleNone.
Units
The poses are plain or physical, all one form. Physical poses (translations in lengths) are read in metres at entry — exactly, the stored value — the solve runs on plain metres, and the calibration comes back physical (Y, X in metres), equal after ustrip(m, ·) to the plain fit's on the poses' ustrip(m, ·). A supplied σ_trans is then a length too (a plain number beside physical poses is a DimensionError). The metadata stay plain numbers whose length is the translation's unit — metres for physical poses: the residuals, the covariance's translation entries, σ_trans. Plain poses give a plain calibration in their own unit, as before.
Inputs must be finite valid rigid transforms (ArgumentError otherwise). Domain-specific gates (depth windows, cheirality) still belong at the call site; a robust optimization_strategy handles the gross poses no gate catches (a sparse, badly registered frame).
See also find_hand_eye for the ratio form A_ij X = X B_ij (no pose prior, RANSAC) which eliminates Y instead of estimating it.
VisualGeometryToolkitEstimation.refine_robot_world_hand_eye — Function
refine_robot_world_hand_eye(guess::RobotWorldHandEyeCalibration{K}, camera::CameraModel, markers;
optimization_strategy, scale | width, max_iter)
-> FitResult{RobotWorldHandEyeCalibration{K}, <:RobotWorldHandEyeMetadata{K, Nothing}}
# defaults: configs/default.toml [refine_robot_world_hand_eye] (the strategy), [refine_camera_and_poses] (the LM)Refine a robot-world calibration A_i X_m = Y B_i^m against the reprojection error of the markers' FEATURES: minimize Σ‖project(camera, Y⁻¹ ∘ A_i ∘ X_m (X)) − u‖² over the shared Y = T_WC and the per-marker X_m = T_RPm, starting at guess (e.g. fit_robot_world_hand_eye's estimate). The camera and the rig poses A are FIXED; nothing is re-estimated from scratch and the family is unchanged — the refine contract.
Robot-world/hand-eye calibration is the literature's name for AX = YB (Zhuang, Roth & Shah 1994; Dornaika & Horaud 1998). Ours is the EYE-TO-HAND configuration — the camera fixed in the world, the targets on the moving rig — with frames in the stack's T_AB notation: Y = T_WC, X_m = T_RPm, A_i = T_WR (the rig pose at frame i) and B_i^m = T_CPm (marker m in the camera at frame i). Function names do not encode frames.
Arguments
guess::RobotWorldHandEyeCalibration{K}— the startingYand the keyedXcamera::CameraModel— the (fixed) camera;project(camera, X_C)maps a camera-frame point to pixelsmarkers::AbstractDict{K, <:AbstractVector{<:Correspondence}}— per marker id, one element per frame it is seen in: that frame's rig poseA_i = T_WR(unlabeled at the boundary) paired with the features seen with it,A_i => [X => u, …]—Xa point in the marker's frame,uits measured pixel. The key sets must match,keys(markers) == Set(guess.ids), and every marker must have at least one feature (ArgumentErrorotherwise): every marker of the guess is a parameter of this refinement and is refined.
Keywords
optimization_strategy— the refine contract's strategy, as everyrefine_*takes it:Triggs{Shape}(the default,Triggs{WelschShape}), ONE robustified LM (Core'sloss_robustifier) fromguesswith no least-squares stage; ornothing, plain least squares. AnIrls{Shape}is anArgumentErrornamingTriggs{Shape}(a refine runs one LM, no reweighting cycles). (local_optimizationis refused, naming this.)scaleorwidth— EXACTLY ONE for a robust strategy, neither for least squares (ArgumentErrorotherwise):scale = σ, the features' noise scale in pixels, or an upstream result whosenoise_scaleit is (the loss width is thenefficiency_constant(Shape, 2) × σ, or the strategy's own constant times σ);width = w, the loss width in pixels. A refine estimates no scale: the metadata records the one it ran with (scale,scale_source,loss_width) and the normalized weights ρ′/ρ′(0) at the estimate;nis their support andrmsthe weighted one,√(Σ wᵢ‖rᵢ‖² / Σ wᵢ).max_iter— the LM's cap on ACCEPTED iterations (NLLSsolver's count; one may hide ~8–10 rejected candidate steps).
Any number of markers: the LM is NLLSsolver's, one pose block per marker and one residual block per point, with nothing compiled per marker count. It is the camera-and-poses bundle's one solve at its settings (refine_camera_and_poses's max_iter, ftol, xtol of configs/default.toml [refine_camera_and_poses], Marquardt's DᵀD damping): this refine has no LM settings of its own. Its termination record (retcode, stop_rule; VisualGeometryToolkitCore's one LM, lmsolve_chart!) is ReturnCode.Success at a tolerance stop (ftol, xtol; Core 0.11.0: whatever the gradient there) or at the rejection cap when the estimate is stationary, MaxIters at the iteration cap (no time cap: it runs without max_time), Failure/InitialFailure (NonFinite) when a solve was non-finite or failed (the estimate is then the last accepted one) and Stalled at the rejection cap short of a stationary point (under a robust strategy the stationarity of the robust cost). Rig poses, points and pixels must be finite, and every point must project at the guess (in front of the camera, inside the lens's domain): an ArgumentError names the first that does not.
The covariance is VGC's implicit_covariance over the reprojection rows at the estimate (each scaled by √ρ′ under a robust strategy) times ŝ² = RSS/dof of those rows — see RobotWorldHandEyeMetadata.
Units
The guess, the rig poses and the marker points are plain or physical, all one form (a mix is a MethodError). Physical ones (lengths) are read in metres at entry — exactly, the stored value — the LM runs on plain metres, and the calibration comes back physical (Y, X in metres), equal after ustrip(m, ·) to the plain refinement's on the inputs' ustrip(m, ·). The residuals and rms are pixels either way; the covariance's translation entries are plain numbers in the translation's unit, metres for physical inputs. Plain inputs give a plain calibration in their own unit, as before.
VisualGeometryToolkitEstimation.RobotWorldHandEyeCalibration — Type
RobotWorldHandEyeCalibration{K, P}
RobotWorldHandEyeCalibration(Y, X::AbstractDict{K, <:EuclideanMap{3}})Result value of fit_robot_world_hand_eye and refine_robot_world_hand_eye (and the guess refine_robot_world_hand_eye takes): the shared camera-in-reference transform Y = T_WC, the per-marker rig-mount transforms X[id] = T_RP(id) keyed by the caller's marker ids, and ids, those keys sorted — the block order of the covariance on the result's metadata. The constructor derives ids from X's keys and stores the transforms unlabeled, all of one form P: plain (Rigid3), or physical — translations in metres — when Y and X are (a mix is a MethodError).
Robot-world/hand-eye calibration is the literature's name for AX = YB (Zhuang, Roth & Shah 1994; Dornaika & Horaud 1998). Ours is the EYE-TO-HAND configuration — the camera fixed in the world, the targets on the moving rig — with frames in the stack's T_AB notation: Y = T_WC, X_m = T_RPm, A_i = T_WR (the rig pose at frame i) and B_i^m = T_CPm (marker m in the camera at frame i). Function names do not encode frames.
VisualGeometryToolkitEstimation.RobotWorldHandEyeMetadata — Type
RobotWorldHandEyeMetadata{K, S, L, W}FitResult metadata of fit_robot_world_hand_eye (S = Float64) and of refine_robot_world_hand_eye (S = Nothing: no pose-residual noise scales; its retcode_symbol(result) is :converged, :max_iterations, :max_time, :solve_failed or :stalled) — one type, assembled in one place. The core carries the RobotWorldHandEyeCalibration (estimate), the retcode, the iterations, the rms and the n observations (the fit: pose observations, its rms over the weighted residual rows; the refinement: points, its rms the reprojection error per point in pixels). Here:
residuals::Dict{K, Vector{Float64}}— per marker, the residual norm of each of its observations in the input order: the fit's weighted 6-residual, the refinement's reprojection error (pixels; element by element, each element'sX => uin order)s2::Float64— the variance factorRSS/dof(≈1 for the fit whenσ_rot/σ_transmatch the data; the pixel variance for the refinement);dof::Int— the rows minus6(M + 1)(a robust estimate's: the weighted rows; with none left,dof = 0,s2 = NaNandcovunbounded)cov::Matrix{Float64}— the joint covariance of the estimate in (rad, translation-unit) tangent coordinates at it, block order[Y | X_{ids…}], each block(δω, δt)in the left-perturbation chartexp(δω)·R, t + δt(VGCRigidExpManifold6). The fit's is first order in the observation noiseŝ²·diag(σ_rot², σ_trans²)through the stages' stationarity conditions (seefit_robot_world_hand_eye); the refinement's iss2 · implicit_covarianceof its reprojection rows.Infeverywhere whenJᵀJis numerically rank deficient (an unobservable direction — seemin_eigval).cov_Y::SMatrix{6,6}— the MARGINAL covariance ofY(a sub-block of the joint covariance — accounts for the X_m coupling)min_eigval::Float64,min_eigvec::Vector{Float64}— the smallest eigenpair of the residual rows'JᵀJ(the same chart ascov): a near-zeromin_eigvalmeans the calibration geometry leaves themin_eigvecdirection unobservable — REPORTED, never hidden.NaNwhen a row is not finite at the estimate (a point the refinement moved out of the lens's domain).σ_rot::S(rad),σ_trans::S— the fit's per-block noise scales of its pose residual (given or profiled at the estimate);nothingfor the refinement, whose residual is pixels at unit isotropic noise.weights::W— the per-datum weights (the fit: one per pose observation; the refinement: one per point), flattened marker by marker inidsorder asresidualsis: lazy unit weights (unit_weights) for a least-squares estimate (L = Nothing,BinaryWeights), the normalized loss weightsρ′of a robust one (Lthe strategy type,IrlsWeights). Read withweights(result);weight_kind(result)dispatches onL.scale::Float64,scale_source::ScaleSource— the scale record of the robust-estimation contract (noise_scale(result),scale_source(result)): the scale a robust estimate ran at and where it came from;NaNandScaleNonefor least squares or a width-only call.loss_width::Float64— the loss width the robust estimate ran at (NaNfor least squares).
Rigid Alignment of Point Sets (ATE)
fit_rigid_alignment is the rigid T with b ≈ R·a + t from point pairs: the alignment an absolute trajectory error is measured after. Least squares (the classical Kabsch on weighted centroids) by default; with optimization_strategy = Irls{K}() and a scale (the trajectory's noise per axis) or a width, the same at one fixed loss width, so a few gross centres do not tilt the alignment.
VisualGeometryToolkitEstimation.fit_rigid_alignment — Function
fit_rigid_alignment(cs; optimization_strategy = nothing, scale | width, tol)
-> FitResult{EuclideanMap{3}, FitMetadata}The rigid transform T (T(a) = R·a + t) with bᵢ ≈ T(aᵢ) from point pairs cs (aᵢ => bᵢ, 3-vectors): the alignment of an estimated trajectory's positions onto the ground truth's for an absolute trajectory error (ATE), or of any two point clouds in correspondence. A fit — it seeds itself with the weighted Kabsch of the centred points (VisualGeometryToolkitCore's kabsch, t from the weighted centroids) — iterating, when robust, a reweighted Kabsch at ONE loss width.
optimization_strategy—nothing(the default: the classical least-squares alignment, one solve), or anIrls{K}strategy (its loss, its efficiency constant and itsmax_iter). A Triggs strategy is refused.scale|width— with anIrlsstrategy, EXACTLY ONE of:scale = σ(the points' noise scale per coordinate, in their unit),scale = result(an upstream robust result'snoise_scale), orwidth = w(the loss width on the residual distance). The width isw, orc · σwithcthe strategy's efficiency constant, by default the shape's at codimension 3 (efficiency_constant(K, 3): Huber 1.628, Welsch 3.4691). Neither, both, or one given to the least-squares fit is anArgumentError.tol— converged when an iteration movesRby less thantolradians andtby less thantoltimes the loss width (default configs/default.toml[fit_rigid_alignment] tol).- A
weightscolumn oncs(withattrs(cs; weights)) multiplies every pair's loss weight (fixed prior weights).
At least three pairs of positive weight are needed, not all on one line (ArgumentError; a robust iteration whose weights leave fewer returns :degenerate). retcode_symbol(result) is :converged, :max_iterations, :solved (least squares) or :degenerate; n is the support (pairs of positive solve weight), rms the weight-weighted root mean square distance — the ATE of the pairs the weighting keeps. Physical points (lengths) are read in metres and the transform comes back physical; the metadata are then in metres.
Its metadata is the reweighted fits' FitMetadata.
Design Document
The full formulation, including identifiability analysis, Tsai–Lenz derivation, decoupled SE(3) residual + Jacobians, and a comparison of loss-function variants, is the Rig-to-Pattern Hand–Eye Calibration guide page; the robot-world form's objective is Robot–World Hand–Eye Calibration.