Radial and Lens Calibration
These estimators calibrate a camera with lens distortion from correspondences between planar calibration boards and their images.
Radial homography
VisualGeometryToolkitEstimation.RadialHomographyProblem — Type
RadialHomographyProblem(pairs; center, sampler=nothing, backend=NativeSolver(),
extra_check=nothing, minimal=:radial)Construct a radially distorted homography problem (one-sided division model on the target side) from 2D-2D point correspondences.
center: the distortion center in TARGET pixels, REQUIRED — the target side is normalized about it (Hartley, mean distance √2). Data, not configuration: the division model is meaningless without its center.extra_check:true(the default when unset) keeps only the cubic root consistent with the redundant 4.5-pt equation per minimal sample (the reference's RANSAC setting);falsescores all real roots. It selects a root of the 5-point kernel, so supplying it withminimal = :homographyis anArgumentErrorrather than a setting that silently does nothing.minimal = :radial: the hypothesis path —:radialsamples five correspondences for the 5-point (H, k) kernel;:homographysamples four for the plain homography, the (H, k) local optimization supplying k on the consensus (header, point 4).pairs(and its:scoreprior)/sampler/backend: asHomographyProblem. The sources areexactor plain and the targets plain (any other covariance is anArgumentError), so the residual is the reprojection error in target pixels: with an exact source and a plain target that is the whitened residual, and its norm the Mahalanobis distance.
VisualGeometryToolkitEstimation.find_radial_homography — Function
find_radial_homography(correspondences; center,
scoring, extent=nothing,
workspace=nothing, extra_check=nothing, minimal=:radial,
local_optimization_strategy, max_trials, min_trials, config=nothing, rng=...)scoring (its width in pixels) and local_optimization_strategy default to configs/default.toml[pipelines.radial_homography], max_trials and min_trials to its [ransac] table; a keyword overrides. A bare threshold or outlier_halfwidth keyword is refused, as find_homography. extra_check is the radial 5-point kernel's root filter (see RadialHomographyProblem) and is refused with minimal = :homography.
Robust radially distorted homography estimation (one-sided one-parameter division model on the TARGET side; Wadenbäck et al. (2026)) via RANSAC with MSAC scoring. minimal = :homography pairs the global search for a plain homography (four-point hypotheses, Homography4pt) with this problem's (H, k) local optimization: the hypothesis carries no distortion term, k is seeded on its consensus by the closed-form least squares in k given H and the joint manifold step estimates both, and the correspondences a plane homography cannot hold under distortion enter at the re-score. The default :radial samples five for the 5-point kernel, whose root pick fails on boards too small to identify k from alone.
Returns a RansacEstimate whose model is a VisualGeometryToolkitCore.RadialHomography in the DATA (pixel) frame — H, k, and the distortion center all carried in the model — the :no_model result (iterations 0) for fewer than a minimal sample (5 correspondences, 4 with minimal = :homography). iterations(result) is the number of trials (hypotheses drawn); test found_model(result) before reading the model (see RansacEstimate).
center — the distortion center in target pixels — is REQUIRED: the 5-pt solver runs in the Hartley frame about it.
local_optimization_strategy defaults to Irls{TLSShape}() (least squares on the inliers) on the 9-dof sl(3) ⊕ k manifold, via the joint 9-dof LM M-step. The chart-rows provider makes LocalOptimizationDoneRight's lifts available on the same manifold.
The other keywords as find_homography.
Units: the source points are plain numbers. A radial homography maps from the source plane, so its entries carry that plane's unit, which a matrix cannot record: physical (unitful) source points are an ArgumentError — strip them in the unit the rest of your plain data uses — metres, as the stack's boundaries strip (ustrip.(m, X)).
VisualGeometryToolkitEstimation.refine_radial_homography — Function
refine_radial_homography(model, correspondences; optimization_strategy, scale | width, workspace = nothing)
-> FitResult{M, RefineMetadata}The radially distorted homography model (VisualGeometryToolkitCore RadialHomography{<:Any, 1}: H, k and the distortion centre, in the data frame; a model with more division coefficients has no method, a MethodError) refined against correspondences from model itself (the reprojection pair, codimension 2). The problem's distortion centre is the model's own (center(model)): the keywords, the result and the refusals are those of every refine_*, documented on refine_homography (default strategy: [refine.radial_homography] optimization_strategy). Fewer than 5 correspondences: the guess back unrefined, record (ReturnCode.InitialFailure, NoRule) (retcode_symbol:degenerate).
Radial camera
VisualGeometryToolkitEstimation.RadialCameraProblem — Type
RadialCameraProblem(pairs; center, sampler = nothing)The radial-camera problem from plane_point => raw_pixel pairs of one board. center — the distortion centre in pixels — is REQUIRED (data, not configuration); a Correspondences:score attribute column drives PROSAC sampling, and every attribute column travels with the data.
VisualGeometryToolkitEstimation.find_radial_camera — Function
find_radial_camera(correspondences; center,
scoring, extent = nothing, workspace = nothing,
local_optimization_strategy, max_trials, min_trials,
config = nothing, rng = ...)Robust estimation of one board's 1-D radial camera (VisualGeometryToolkitCore RadialCamera: rows 1–2 of the plane homography, invariant to focal length and to any radially symmetric distortion about the centre) from plane_point => raw_pixel correspondences, via RANSAC on the linear 5-point solve with MSAC scoring on the point–radial-line distance. Returns a RansacEstimate whose model lives about the distortion centre center (pixels, REQUIRED), the :no_model result (iterations 0) for fewer than 5 correspondences. iterations(result) is the number of trials (hypotheses drawn); test found_model(result) before reading the model (see RansacEstimate).
The uncalibrated verification model for board correspondences: no distortion parameter is estimated per board (that is the focal stack's job, across boards), so the solve is closed form with no root to pick. scoring (its width in pixels) and local_optimization_strategy default to configs/default.toml[pipelines.radial_camera] — MSAC, and least squares on the inliers, Irls{TLSShape}() — and the trial bounds to its [ransac] table. The other keywords as find_homography.
Units: the source points are plain numbers. A radial camera maps from the source plane, so its entries carry that plane's unit, which a matrix cannot record: physical (unitful) source points are an ArgumentError — strip them in the unit the rest of your plain data uses — metres, as the stack's boundaries strip (ustrip.(m, X)).
VisualGeometryToolkitEstimation.refine_radial_camera — Function
refine_radial_camera(model, correspondences; optimization_strategy, scale | width, workspace = nothing)
-> FitResult{M, RefineMetadata}The 1-D radial camera model (VisualGeometryToolkitCore RadialCamera, about its distortion centre) refined against plane_point => raw_pixel correspondences from model itself (the pixel distance to the predicted radial line; codimension 1). The problem's centre is the model's own (center(model)): the keywords, the result and the refusals are those of every refine_*, documented on refine_homography (default strategy: [refine.radial_camera] optimization_strategy). Fewer than 5 correspondences: the guess back unrefined, record (ReturnCode.InitialFailure, NoRule) (retcode_symbol:degenerate).
VisualGeometryToolkitEstimation.fit_radial_camera — Function
fit_radial_camera(cs::AbstractVector{<:Correspondence{<:StaticVector{2}, <:StaticVector{2}}};
center, iterations = 2)
-> FitResult{RadialCamera{Float64}, Nothing}The radial camera of one board from its plane_point => image_point correspondences cs, the image points RAW as measured; center (pixels, REQUIRED — the principal point, as find_radial_camera and RadialCameraProblem require it) is the pencil's centre and rides on the returned camera, so nothing is pre-centred. There is no default: a pencil about the image corner fits raw pixels as well as any other, with no error. Each pair gives one linear constraint on the six entries of P (constraint_with_jacobians); the null vector is solved by SVD in the radial camera's normalization, CorrespondenceNormalization(RadialCamera, cs; center) (Hartley at the plane centroid, Hartley about center on the image side — the same strategy find_radial_camera's problem solves in), and composed back into pixels. Each iteration re-solves; from the second on every row is divided by the constraint's observation-gradient norm at the previous iteration's solution (whitening_factor, the Sampson normalizer held fixed — the algebraic error becomes the exact point–radial-line distance); a weights column on cs (a Correspondences attribute, e.g. withattrs(cs; weights)) multiplies on top, the row weight being √w. iterations = 1 is the plain algebraic (unwhitened) solve. This is how the BabelCalib back-end obtains the radial camera when no homography is available; when one is, its rows 1–2 are the same object (RadialCamera(H)). Five correspondences determine P (five degrees of freedom).
The FitResult's estimate is the camera, its record (ReturnCode.Success, ClosedForm) after iterations iterations (its iterations field), and rms the point–radial-line distance (px) of the n pairs at the fit, whatever the iteration count. When the constraint rows leave more than one null direction (a degenerate configuration — the plane points on a line, or a pixel on the centre) the record is (ReturnCode.InitialFailure, NoRule), the camera NaN and the rms NaN.
Units: the source points are plain numbers. A radial camera maps from the source plane, so its entries carry that plane's unit, which a matrix cannot record: physical (unitful) source points are an ArgumentError — strip them in the unit the rest of your plain data uses — metres, as the stack's boundaries strip (ustrip.(m, X)).
Division camera
fit_division_camera fits a camera with a division lens model to several board views at once. division_camera_step is the model update it repeats under its current weights. The robust loop is the stack's shared one: the posterior of a StudentTUniformScoring (the weighting keyword) on the posterior-weight iteration of a MixtureEmConfig (the mixture_em keyword), over the stack as a DivisionCameraProblem.
VisualGeometryToolkitEstimation.fit_division_camera — Function
fit_division_camera(radial_cameras::AbstractDict{K}, views::AbstractDict{K};
image_size = nothing, focal_seed = nothing, ncoef, k = nothing,
weighting, mixture_em, # defaults: configs/default.toml [fit_division_camera]
verbose = false)
-> FitResult{CameraModel, DivisionCameraMetadata}The BabelCalib linear focal solve as an iterated, gradient-whitened, robustly re-weighted estimator (see the file header). Keyed by the view's identity K (the key sets must be equal — ArgumentError): radial_cameras[id] is view id's radial camera (a RadialCamera, or a homography whose rows 1–2 are read) and views[id] its plane_point => raw_pixel correspondences — per-board VALIDATED inliers (the cameras carry the pencil's shared centre) — the inputs of division_camera_step. A view's weights column, if it has one, is a prior row weight: iteration 0 is weighted by it and each later iteration by it times the posterior. k fixes the division coefficient (centred pixels); otherwise ncoef are estimated. Each iteration whitens every row at the previous solution (the row's Sampson factor held fixed) and weights it by the Student-t/uniform posterior of the previous iteration's whitened residuals.
THE ROBUST MACHINERY IS THE STACK'S SHARED ONE (configs/default.toml [fitdivisioncamera], one schema):
weighting— the E-step: the posterior of aStudentTUniformScoringat codimension 2 with a supplied outlier half-width, its scale prior the posterior's (default the table'sweighting, read byscoring_from_tomlwithproblem = DivisionCameraProblem; absent keys inherit [scoring.studenttuniform]). Another codimension, or one without a half-width, is anArgumentError. A fit takes noscoring(the robust-estimation contract): the formerscoringkeyword is anArgumentErrornamingweighting.mixture_em— the loop: aMixtureEmConfigdriving the posterior-weight iteration (PosteriorIrlsMethod) — its model update must beWeightedRefine(the stack is refitted each iteration), its stopParameterChangeStop(each iteration re-whitens at the previous solution, so no cost is comparable between iterations: the fit stops when (f, k) moves by less thanrtolrelative and returns the LAST iterate; the incumbent rule stopped the contaminated stacks at their second iteration, f 2 % off), its nuisance policies the posterior's and itsmax_iterthe budget (default the table'smixture_em, read bymixture_em_from_toml; absent keys inherit [mixture_em]). Anything else is anArgumentError.
outlier_halfwidth and max_iter are no keywords since 0.9 (an ArgumentError naming the form above): the half-width is the weighting's, the cap the EM's.
INPUT CONTRACT: each view is that board's VALIDATED correspondences — the inliers its radial camera was found with — not raw detections. The Student-t/uniform posterior absorbs what validation leaves (a few per cent of ray-aligned mismatches, which a radial camera's 1-D line test cannot reject); it is not a substitute for it.
THE BRANCH AUDIT. Each view's pose branch — the tilt ambiguity of its radial camera — is chosen on the iteration that first places the view, from that view's own solve, and kept. That solve cannot separate the tilt pair by residual, so one mismatch pushed along its ray through the centre can tip it (on the MC scene at 3 % ray-aligned mismatches, 16 % of the views carrying one took a branch 48–85° off; none of the clean views), and the view's inliers then carry residuals far above sigma — after the posterior has already rejected the mismatch. So once the IRLS converges — or, if it is still moving at half the mixture_em.max_iter budget, there (a wrong branch is what keeps a stack from converging) — one iteration re-judges every view's branch at the shared (f, k), robustly; if it moves any, the IRLS resumes from there within what is left of max_iter (the audit counts as an iteration, so iterations never exceeds max_iter); if it moves none the IRLS continues unchanged, bit for bit (clean stacks). Never after :solve_failed, nor at max_iter = 1. Measured: 104 → 4 wrong views at 3 %, 66 → 1 at 5 %, no correct view moved, and the held-out residuals calibrated against ŝ²·I₂ in the tail as well as the bulk (test/ransac_mc_calibration_tests.jl (e)).
Inputs, k, the scorer's outlier_halfwidth, focal_seed and every output are in pixels; the stack is conditioned inside by one normalization (CorrespondenceNormalization(DivisionCameraState, views; center) — Hartley about the radial cameras' shared centre), built once. image_size = (width, height) (pixels) of the image the views were measured in is REQUIRED for the default seed only (ArgumentError when both it and focal_seed are missing): its half-diagonal s = hypot(width, height)/2, a 90° diagonal field of view. With focal_seed given, image_size may be omitted and does not enter the result.
Iteration 0 has no previous solution: it is whitened at a SEED camera — focal focal_seed (pixels; by default s), k₀ = 0 (the given k when k fixes it — the known-k rows are exact only there), each board at the depth that camera implies — which sets the row WEIGHTS only; f and k are solved by the iteration. The plain algebraic rows it replaces weigh each board by its depth (a far board's rows ~10× a near one's), so a few mismatches on a heavy board dragged the joint solve to f ≈ 45–300 px against 1226 and half those stacks never recovered; the seed fixes that and is robust to a seed 0.5×–3× off (measured 2026-09-22; see test/ransac_mc_calibration_tests.jl (e)). Supply a better focal as data — a previous calibration, EXIF — through focal_seed.
The scorer's outlier_halfwidth (pixels) is the half-width of the uniform outlier component — the same quantity the RANSAC problems take; it must NOT be the residual scale (a half-width at the median residual makes the outlier component as dense as the inliers and zeroes every weight — measured). The posterior is seeded from the median residual (median_seeded_posterior): every point starts an inlier and the M-step shrinks ŝ as the model improves.
That seed and schedule are load-bearing, not incidental. Five alternatives were measured against RVG's synthetic (known f₀, 0.4 px noise, 8 % gross outliers) on 2026-09-10 and every one was worse: a frozen MAD anchor 6.5 % off f₀, a global Jeffreys sweep 3.5 %, a per-board sweep 5.1 %, frozen Welsch 28 %, Welsch with an adaptive scale 85 % (no uniform component ⇒ the scale update has no fixed point and ran to ŝ = 29). This estimator lands at 1.25 %. A sweep seed needs residuals that are already bimodal; iteration 0 here is one linear solve of an uncalibrated stack (whitened at a seed, not at a fit), so they are not. (Those numbers predate the seeded iteration 0.)
Iteration 0 alone — whitened at the seed and robustly unweighted — is the step
N = CorrespondenceNormalization(DivisionCameraState, views; center) # center: the cameras' shared centre
CameraModel(division_camera_step(N, radial_cameras, views; ncoef, k, whiten_at = focal_seed))(with focal_seed the half-diagonal by default); this function always runs at least one robust iteration (mixture_em.max_iter ≥ 1), and max_iter = 1 is that step re-weighted once, with no branch audit. A stack whose iteration 0 cannot solve at all (no board admits a pose, none carries depth, the solve is singular) raises the DivisionCameraError — there is no iterate to return; a later iteration that cannot solve ends the fit on the last good one (:solve_failed).
The radial cameras are FIXED. Each view's radial camera is an input, estimated once by the caller — BlobBoards fits every one linearly about the centre, iterated to its consensus and polished by a local optimization on the exact point–radial-line distance — and this stage only turns them into an initial focal length, coefficients and depths for a bundle. It used to re-fit them per iteration under the point weights, which could not improve a nonlinear fit and did wreck one: the perturbation tipped the metric upgrade's cheirality test onto the mirrored branch for a board the shared solve had placed behind the camera, turning its pose 179.95° and its whitened residual from 1.7 px to 1102 px, and through the Student-t width that single row collapsed every weight in the solve — 18 of 106 boards lost their poses, boards whose own poses under the converged bundle camera were the BEST in the set at 0.14 px (measured 2026-09-09, fisheye hallway). Hand this function cameras you are willing to stand behind; it will not launder contaminated ones.
Returns the camera CameraModel(::DivisionCameraState) builds — K = diag(f, f, 1) at the cameras' shared centre, Division{ncoef} lens in K's chart — plus the final keyed DivisionCameraState (pixels), the views with each correspondence's IRLS row weight as their weights column, and the loop outcome — and the robust-estimation contract's record of the E-step (DivisionCameraMetadata): weights(result), each correspondence's posterior inlier probability (PosteriorWeights: inlier_mask(result) is posterior > ½), and noise_scale(result), the E-step's Student-t scale ŝ (pixels, a LINEAR scale; scale_source(result) is ScaleEstimated), with its ν. n is the support (the rows of positive weight) and rms the posterior-weighted whitened-residual RMS, √(Σwᵢrᵢ² / Σwᵢ) (pixels): a row with prior weight zero, or of a view the final iteration left without a pose, weighs 0 and is not a residual.
Units: the board points are plain numbers. The radial cameras and the board poses carry the board plane's unit, which neither a matrix nor this result records, so physical (unitful) board points are an ArgumentError: strip them in the unit the radial cameras were fit in (the stack's own boundaries strip in metres, ustrip.(m, X)).
VisualGeometryToolkitEstimation.DivisionCameraMetadata — Type
DivisionCameraMetadataFitResult metadata of fit_division_camera, keyed by the views' identity: the converged solve state (a DivisionCameraState{K} — f, the BabelCalib coefficients, the per-view poses), the radial_cameras the fit consumed, the views carrying each correspondence's final IRLS row weight as their weights column (the posterior's weight normalized to max 1, times the input column when the view had one — the weight the solve ran with), and the robust-estimation contract's record of the final E-step:
weights, every correspondence's posterior inlier probability, flat in the views' key order (sort(collect(keys(views))), each view's in its order) —PosteriorWeights, soinlier_mask(result)is the MAP rule (posterior > ½); a row the fit did not take (prior weight zero, or a view the final iteration left without a pose), and a posterior beloweps(), is an exact 0; read byweights(result);scale= ŝ (pixels, a LINEAR scale) withscale_sourceScaleEstimated, read bynoise_scale(result); and the Student-tν.
The core fields carry the camera (estimate), retcode, iterations, the support n (the rows of positive weight) and the whitened-residual rms weighted by those weights, √(Σwᵢrᵢ² / Σwᵢ) (pixels).
VisualGeometryToolkitEstimation.DivisionCameraProblem — Type
DivisionCameraProblemThe stacked division-camera solve as a problem of the shared posterior-weight iteration (PosteriorIrlsMethod under WeightedRefine): the views' radial cameras and board points, their prior row weights, the coefficient mode and the stack's one normalization, built by fit_division_camera. Its distances (mahalanobis_distances!) are the whitened residual norms of the correspondences (pixels, codimension 2) and its refine! is one weighted, whitened division_camera_step. As a TYPE it is what the fit's weighting (its Student-t + uniform E-step) is built for: scoring_from_toml(table; problem = DivisionCameraProblem) or StudentTUniformScoring(DivisionCameraProblem, h; scale_prior, label_prior) (codimension 2).
VisualGeometryToolkitEstimation.division_camera_step — Function
division_camera_step(N, radial_cameras::AbstractDict{K}, views::AbstractDict{K};
ncoef = 1, k = nothing, whiten_at)
-> DivisionCameraState{K}A declared INNER-LOOP piece (public, not exported): it takes the estimator's normalization N and pixel data, and returns pixel state. N is REQUIRED — CorrespondenceNormalization(DivisionCameraState, views; center), the one strategy fit_division_camera builds — applied to the data inside, never built here; there is no fallback.
One iteration of the stacked linear focal solve via the division-model lift (BabelCalib's back-end back-substitution, Lochman et al. (2021)), as a DivisionCameraState: the shared unknowns plus the per-board poses and the whitened per-point residuals the next iteration needs. Square pixels are assumed. Both inputs are keyed by the view's identity K (the key sets must be equal — ArgumentError): radial_cameras[id] is view id's radial camera (a RadialCamera, or a HomographyMat whose rows 1–2 are read) in the SAME plane frame as views[id], that view's plane_point => image_point correspondences with the RAW (distorted) image point in pixels. The pencil's centre rides on the cameras, must be shared across the stack, and is carried by the result; a bare HomographyMat means the origin.
Per board, the radial camera fixes the rotation and (t₁, t₂) up to four sign candidates; the remaining unknowns are LINEAR in the division-model lift (x, y, 1 + Σₚ kₚ r²ᵖ) ∝ (X, Y, Z + t₃) (RadialLift). The candidate is chosen by cheirality and residual, and every board is stacked into one least-squares system with the shared unknowns and one t₃ per board. A view's weights column (a Correspondences attribute, e.g. withattrs(cs; weights)) weighs its rows — a board-level vote folds into it losslessly, the row weight being √w; views with no admissible candidate are dropped, and a DivisionCameraError is thrown if none survive or the stack yields f ≤ 0 (the argument contract still raises ArgumentError). With k given (the one-parameter division coefficient in the centered-pixel frame, e.g. RadialHomography.k), only f and the t₃'s are estimated (ncoef must stay 1); otherwise the stack carries ncoef ≥ 1 extra shared columns λₚ = kₚ·f and estimates the coefficients of r², r⁴, … too.
N is the normalization the stack is solved in: its image side must be an isotropic scale about the radial cameras' shared centre, its plane side an isotropic similarity (ArgumentError otherwise). The state is returned in pixels regardless. whiten_at whitens every row (the rows' Sampson factor L⁻¹ of G_x·G_xᵀ, whitening_factor, frozen there) at
- a previous iteration's
DivisionCameraState— its solution, each view in the branch that iteration chose; or - a SEED focal length (a positive
Real, data units): the pinholef, k₀ = 0 (the givenkwhenkfixes it), each view in its freshly chosen branch at the depth that camera implies. The seed sets the WEIGHTS only —fandkare still solved by this iteration. It puts iteration 0 in pixel currency, where the plain rows weigh each view by its depth (a far board's rows ~10× a near one's); seefit_division_camera.
The camera an iteration determines is CameraModel(state): K = diag(f, f, 1) with the principal point at the cameras' shared centre, and a Division{ncoef} lens re-expressed in that K's normalized frame (kₚ,norm = kₚ,px · f^(2p), VGC's radial-coefficient law — the lens law of the CameraModel(model, K′) constructor).
ONE ITERATION, not an estimator. Without whiten_at there is nothing to linearize at, so the SOLVE is plain algebraic least squares; the returned sq_distances, which the robust driver reweights by, are the Sampson residuals at the solution. The robust driver's iteration 0 is this step at its seed,
N = CorrespondenceNormalization(DivisionCameraState, views; center)
CameraModel(division_camera_step(N, radial_cameras, views; ncoef, whiten_at = focal_seed))(fit_division_camera builds the same normalization). For a converged estimate call the robust driver.
Units: the board points are plain numbers. The radial cameras and the board poses carry the board plane's unit, which neither a matrix nor this result records, so physical (unitful) board points are an ArgumentError: strip them in the unit the radial cameras were fit in (the stack's own boundaries strip in metres, ustrip.(m, X)).
VisualGeometryToolkitEstimation.DivisionCameraState — Type
DivisionCameraState{K}Result of one iteration of the stacked linear focal solve, in DATA (pixel) units, keyed by the views' identity K: f; the division coefficients k (of r², r⁴, … in pixels centred on centre); the pencil's centre, the principal point every input radial camera shares; poses[id], the board pose — an EuclideanMap{3} from the board plane (z = 0) into the camera, rotation from the radial camera, translation (t₁₂, t₃) — in the plane's units, of each view that admitted one (a view without a pose this iteration is absent); and sq_distances[id], the whitened residual norms² of the view's correspondences (pixel², in its order; NaN for a view without a pose) — the currency the robust re-weighting reads. ids is the sorted key order. Feed it back as whiten_at for the next iteration; CameraModel(state) is the camera it determines.
VisualGeometryToolkitEstimation.DivisionCameraError — Type
DivisionCameraError(msg)The stacked division-camera solve (division_camera_step) has no system to solve for the boards it was given: no board admits a pose with its points in front of the camera and imaged on the same side of the centre, no board carries depth information (every row weighted to zero or at the centre), the shared unknowns are not determined or not finite, or the focal length comes out non-positive. A condition of the DATA, not of the arguments — the argument contract (one correspondence set per camera, one shared centre, …) still raises ArgumentError — so a caller that can go on without the solve catches this type and nothing else.
VisualGeometryToolkitEstimation.RadialLift — Type
RadialLift{N,T}The division-lift constraint model of ONE board: the partial pose (R, t12) from its radial camera and the linear unknowns θ = (f, λ₁..λ_N, t₃) — f, the lift coefficients λₚ = kₚ·f of r²ᵖ, and the board's depth. For a plane point X and its RAW centered pixel x the constraint is
g = x·(Z + t₃) − X_c·(f + Σₚ λₚ ρ²ᵖ) (two rows; X_c = camera-frame point at t₃ = 0)which is LINEAR in θ: g = G_θ·θ + c₀, c₀ = x·Z. constraint_with_jacobians returns g, the observation Jacobian G_x = ∂g/∂x (2×2) and G_θ (2×(N+2)), and its frozen whitening factor (whitening_factor, L⁻¹ of G_x·G_xᵀ) turns them into pixel-currency rows. A KNOWN k folds λ₁ = k·f into the f column (chain rule) — see _lift_columns.
VisualGeometryToolkitCore.division_coeffs — Method
division_coeffs(state::DivisionCameraState) -> Vector{Float64}The iteration's division coefficients k (of r², r⁴, … in pixels about the centre): VisualGeometryToolkitCore's division_coeffs, answered for the state.
VisualGeometryToolkitEstimation.board_sq_residuals — Function
board_sq_residuals(state::DivisionCameraState{K}) -> Dict{K, Vector{Float64}}The whitened residual norms² of every correspondence, per view in its order (pixel²) — the currency the robust re-weighting reads.
Camera and poses
refine_camera_and_poses refines a camera — its intrinsics and lens, on VisualGeometryToolkitCore's ParameterManifold — jointly with one pose per view (LeftRigidManifold6) against the reprojection error of 3D → 2D correspondences, from a guess of both: the bundle of a calibration with a known target, for any lens the camera's family holds (a fisheye's rays past 90° are ordinary data). Naming every parameter in fixed holds the camera and refines the poses alone. Its strategy is the refine contract's optimization_strategy — Triggs{WelschShape}() by default, run from the start with no least-squares stage, at the caller's scale = σ (a refine estimates no scale) or width = w; nothing is plain least squares. A SplineLens camera is refined only by back-projection (fit_camera(SplineLens, …), pspline_grid), or held: here it takes fixed = (:controls,), the poses refined through its whole project under the bundle's strategy and the camera returned as given (its forward spline kept); the image-space refine of its controls was retired in 0.10.0 (the default fixed = () and a penalty are refused, naming it). With several rigid boards, refine_camera_and_poses(camera, poses, board_poses, views) adds one pose per board (board → world) to the same bundle, the correspondences keyed by (view, board); each connected group of views and boards keeps its gauge by a held pose (its first board by default, held_boards/held_views), exactly.
refine_camera_and_poses(camera, views::RotationCenterViews, correspondences) refines the views of a camera turned about one fixed point (a tripod head) under one rigid motion model, one rotation per view and a shared centre and offset, the camera held. Its residual is each dot's BEARING for every camera: the dot back-projected once by the camera, the tangent angle to the predicted bearing whitened into pixels by the back-projection Jacobian (the BearingPoseProblem's); the camera is never projected inside the solve, and a dot it does not back-project, or whose Jacobian cannot whiten its residual (pixel_whitening_defect), makes no row and is counted in the metadata (n_without_bearing, without_bearing). Its fixed holds the whole camera: fixed = held_parameters(camera). Its fourth argument, the residual type, is BearingResidual() by default; ReprojectionResidual() adds a second stage from the bearing estimate on the exact pixel reprojection error (a SplineLens camera projected by inverting its backward spline), a candidate update that leaves a prediction undefined rejected by the LM, the dots that do not project at the bearing estimate counted (without_projection).
VisualGeometryToolkitEstimation.refine_camera_and_poses — Function
refine_camera_and_poses(camera, poses, views; fixed = (), box = nothing, optimization_strategy, scale | width,
max_iter, ftol, xtol, max_time = nothing) # defaults: configs/default.toml [refine_camera_and_poses]
refine_camera_and_poses(spline_camera, poses, views; fixed = (:controls,), # required: a SplineLens camera is held
optimization_strategy, scale | width, max_iter, ftol, xtol, max_time = nothing)
-> FitResult{<:NamedTuple{(:camera, :poses)}, CameraPosesRefineMetadata}
refine_camera_and_poses(camera, poses, board_poses, views; held_boards, held_views = (), fixed = (), box,
optimization_strategy, scale | width, max_iter, ftol, xtol, max_time = nothing)
-> FitResult{<:NamedTuple{(:camera, :poses, :board_poses)}, CameraPosesRefineMetadata}Refine a camera and one pose per view jointly against the reprojection error of 3D → 2D correspondences: minimize Σ ‖project(camera, pose_v(X)) − u‖² over the camera's free parameters and every view's pose, from the guesses camera and poses. A refine: both guesses are required, nothing is re-estimated from scratch and the camera keeps its family (its lens type).
Arguments
camera— the camera guess, aCameraModelor aBoundedCameraof one (the estimate is then bounded alike): intrinsics and lens, the parameters VisualGeometryToolkitCore'sparameter_vector(camera)lists (f, aspect, skew, cx, cy, then the lens's coefficients), in pixels. A SplineLens camera is held, its poses refined (below).poses— one world → cameraEuclideanMap{3}per view (X_c = R·X + t,Xin the target's frame), keyed likeviews.views— per view, its correspondencescs = [X => u, …]:Xa target point (3-vector),uthe pixel it is detected at (2-vector). Grouped data: anAbstractDict{K, <:AbstractVector{<:Correspondence}}keyed by the view, withposes::AbstractDict{K}of the same keys (block order: the sorted keys; anArgumentErrorwhen the key sets differ), or a vector of views with a vector of poses of the same length. Every point must project at the guess (in front of the camera, inside the lens's domain): anArgumentErrornames the first that does not.
Keywords (configuration)
fixed— the camera parameters HELD, by name or by group (:kfork1…; VisualGeometryToolkitCore'sfree_parameter_namesresolves it),()the default: every parameter is refined but the gauge, the skew, which is held by construction (naming it is accepted); naming every parameter holds the camera and refines the poses alone. A SplineLens camera takes(:controls,)only (below). The stack's one parameter-holding keyword (NLLSsolver's word; OpenCV'sCALIB_FIX_*):free,fix,fixed_paramsandholdare refused, naming it.box— a solver search box on the camera's parameters by name (VisualGeometryToolkitCore'sParameterManifold(camera; box)), ornothing; with every parameterfixed(nothing to bound) a box is anArgumentError.optimization_strategy— the refine contract's strategy, as everyrefine_*takes it:Triggs{Shape}(the default,Triggs{WelschShape},configs/default.toml [refine_camera_and_poses]), one robustified LM (Core'sloss_robustifier), starting directly from the caller's start (camera,poses, board poses): it runs no least-squares stage, so gross outliers never pull a warm-up fit; ornothing, plain least squares (it reads no scale). TOML:{ type = "triggs", loss = "welsch" }. AnIrls{Shape}is anArgumentErrornamingTriggs{Shape}(a refine runs one LM, no reweighting cycles); any other strategy is one too (it needs a RANSAC consensus set). (The keyword wasrobust, thenlocal_optimization; both are refused, anArgumentErrornaming this one.)scaleorwidth— EXACTLY ONE for a robust strategy (so for the default), neither for least squares (ArgumentErrorotherwise; Core'sscale_input):scale = σ, the detections' noise scale (pixels, per coordinate), or an upstream result whosenoise_scaleit is; the loss width is then the strategy's efficiency constant times σ (Core'sefficiency_constant(Shape, 2)unless the strategy states one).width = w: the loss width itself (pixels). A refine estimates no scale; the metadata records the one it ran with (scale,scale_source,loss_width) and the weights ρ′/ρ′(0) at the estimate.max_iter— the LM's cap on ACCEPTED iterations (NLLSsolver's count);ftol— its relative cost decrease;xtol— its step tolerance (chart units);max_time— its wall-clock cap in seconds (nothingorInf: none), for a camera whose projection makes an iteration slow; it stops with:max_time.
The estimate is (; camera, poses): the refined camera, and the refined poses keyed like the input (a Dict by the same keys, or a vector in view order, each with its input's frame labels). Its termination record (retcode, stop_rule) is the last solve's, VisualGeometryToolkitCore's one LM (lmsolve_chart!): ReturnCode.Success at a tolerance stop (ftol, xtol; Core 0.11.0: whatever the gradient there) or at the rejection cap when the estimate is stationary, else MaxIters (the iteration cap), MaxTime, Failure/InitialFailure (NonFinite) or Stalled (the rejection cap short of a stationary point); rms is √(Σ‖rᵢ‖² / n) over the n correspondences (pixels). The metadata (CameraPosesRefineMetadata) carries the residual rows, s², ν and the free parameter names. Plain least squares: the correspondences must be inliers.
Several boards
With board_poses, the target is several rigid boards, each with its own pose: poses maps each view to its world → camera pose, board_poses each board to its board → world pose (both AbstractDicts of plain EuclideanMap{3}), and views maps (view, board) to the correspondences X => u of that board's points (X in the board's frame) detected in that view. The residual is project(camera, pose_v(board_b(X))) − u; each board pose is one more pose chart of the same NLLSsolver problem, beside the camera's chart and the view poses (the same assembler and LM, the same retcode and metadata; ν counts 6 per free board too). A pose the data never reach is an ArgumentError, and so is a correspondence of a view or board without a pose.
The gauge: a common rigid motion of every board, undone by every view, leaves the cost unchanged, once per connected component of the view–board graph (views and boards linked by their correspondences). It is fixed EXACTLY, by holding poses at their guesses (no pin, no prior): held_boards (by default each component's first board, in sorted key order) and held_views (none by default) are held, every other pose is a variable, and a component without a held pose is an ArgumentError. Holding more than one pose of a component constrains the solve (a held pose is known). The estimate's board_poses returns a held pose as given. A parametric camera only (a SplineLens camera's gauge is its own: an ArgumentError); plain poses and points.
A SplineLens camera
Its K is a frozen gauge (skew 0) and its parameters are its backward lattice's controls, so its route differs (Core's IntrinsicsRole trait, GaugeIntrinsics, selects it). A SplineLens camera is refined only by back-projection, or held: its backward is fitted on back-projected rays (fit_camera(SplineLens, …), pspline_grid), and here it is held while the poses are refined.
fixed = (:controls,)is required: the camera held, the poses refined. The defaultfixed = (), which would refine the controls against the pixel rows, is anArgumentErrornaming the retired image-space refine (VisualGeometryToolkitEstimation 0.10.0), and so is apenalty(that route's smoothness weight); aboxand any otherfixedareArgumentErrors.- Each correspondence's rows are
project(camera, X_c) − uthrough the whole held camera, a constant (the parametric route's block, on its pose alone): no window, no node column. A correspondence detected on a window with an absent node is dropped and counted (result.spline.n_dropped_window), and so is one the camera does not project when detected within one knot cell of such a window (it images it onto the absent nodes; counted apart,result.spline.n_dropped_unprojected: it can be an inlier); every other one must project at the guess (anArgumentErrornames the first that does not, or where the lattice folds). - The camera comes back AS GIVEN, its forward spline included (none is refitted): its controls are no variable, so there is no gauge to fix and no frame to turn, and the poses come back in the camera's own frame. A bare model or a
BoundedCameraalike. Each row is the backward's own projection solve (a forward the camera holds is not used in the solve: it is the backward's approximation). The keywordsrefit_forwardandforward_toleranceof the retired forward refit are refused by name (0.11.0); give a camera its forward withfit_forward_spline. - The strategy is the bundle's (
optimization_strategy, itsscaleorwidth: the default Triggs{Welsch} needs one;nothingis least squares).max_timecaps the solve (the record(ReturnCode.MaxTime, TimeCap)). - Device: the projections that check the guess run batched on the device (Core's
default_device()); the LM is NLLSsolver's, on the CPU.
Units
Plain inputs give plain poses; physical ones (unitful poses and target points, all one form) are read in metres at entry and the poses come back physical, in metres. The residuals are pixels either way.
refine_camera_and_poses(camera, views::RotationCenterViews, correspondences, residual = BearingResidual(); fixed, optimization_strategy, scale | width,
max_iter, ftol, xtol, max_time = nothing)
-> FitResult{<:NamedTuple{(:camera, :views)}, RotationCenterRefineMetadata}The bundle under the rotation-centre view parametrization: the views of a camera turned about one fixed point (RotationCenterViews: per view its rotation R_k, shared by all the centre c and the offset d, view k's pose (R_k, d − R_k c)), refined from the guess views against correspondences — per view K its correspondences X => u (target point, pixel; the key sets must match). The residual is the BEARING residual for every camera: each dot back-projected once by the camera (backproject_with_jacobian) to its unit bearing, the residual the tangent-plane angle to the predicted bearing of R_k (X − c) + d, whitened into pixels by the dot's back-projection Jacobian (the BearingPoseProblem's, bearings.md) — so rms, residuals and widths are pixels, and no point is projected through the camera inside the solve. A dot the camera does not back-project (no finite bearing), or whose back-projection Jacobian cannot whiten the residual (not finite, not tangent, singular), makes no row and is counted: n_without_bearing and without_bearing (its (view, correspondence index) pairs). The variables: one rotation chart per view and the centre and the offset; the camera is HELD — fixed must name every parameter of a parametric camera (fixed = (:controls,) for a SplineLens camera, its route's spelling): held_parameters(camera) is that fixed; a free camera is an ArgumentError. The keywords, the solve, the retcode and its stop rule, rms, n and the robust record are the bundle's (ν = 2n − 6 − 3N; scale = result records ScaleUpstream).
The gauge: none when the views' relative rotations span two axes. Every relative rotation about one axis (or fewer than three views) leaves the centre free along it against the offset, and is an ArgumentError.
The estimate is (; camera, views): the camera as given, and the refined RotationCenterViews keyed like the input.
residual (the fourth argument): BearingResidual(), the default, the bearing residual above; ReprojectionResidual() that refine, then a second stage from its estimate on the exact pixel reprojection error (the result's views, retcode and record are the second stage's).
VisualGeometryToolkitEstimation.CameraPosesRefineMetadata — Type
CameraPosesRefineMetadataFitResult metadata of refine_camera_and_poses: residuals, the reprojection rows at the estimate (pixels, two per correspondence: views in block order, each view's correspondences in their order); s², the residual variance Σ‖rᵢ‖² / ν (px²); ν = 2n − p, its degrees of freedom (p the free camera parameters plus 6 per view); free, the names of the camera parameters refined (() for a SplineLens camera, which is held); and spline, nothing for a parametric camera, for a SplineLens camera (; n_dropped_window, n_dropped_unprojected): the correspondences detected on an absent window, and those the camera does not project when detected within a cell of one (they can be inliers). For a SplineLens camera p = 6·views (the camera is held). Then the robust record: residual_norms (one per correspondence, pixels), weights (the loss's ρ′/ρ′(0) at the estimate under a robust strategy, all ones — lazily — under least squares), and the scale the loss ran at, scale and scale_source (ScaleGiven/ScaleUpstream, or NaN/ScaleNone under least squares and for a width given alone; noise_scale reads it) and loss_width (NaN under least squares). All forward as properties: result.residuals, result.spline, ….
VisualGeometryToolkitEstimation.RotationCenterViews — Type
RotationCenterViews(rotations, center, offset)The views of a camera turned about one fixed point (a tripod head) for refine_camera_and_poses: rotations maps each view K to its rotation R_k (target → camera), center is the rotation centre c (target frame) and offset the projection centre's offset from it, d (camera frame, shared by every view): view k's pose is X ↦ R_k (X − c) + d, i.e. (R_k, d − R_k c). Plain numbers (the target's length unit).
VisualGeometryToolkitEstimation.rotation_center_pose — Function
rotation_center_pose(R, center, offset) -> EuclideanMapThe pose (target → camera) of one view of RotationCenterViews: the camera turned by R (target → camera) about the rotation centre center (target frame), its projection centre at offset from it (camera frame), so X ↦ R (X − center) + offset, i.e. EuclideanMap(R, offset − R·center). The pose the rotation-centre refine evaluates inside its bearing block, and the one a caller builds from a result's views: rotation_center_pose(v.rotations[k], v.center, v.offset) with v = estimate(result).views. Plain numbers (the target's length unit).
VisualGeometryToolkitEstimation.held_parameters — Function
held_parameters(camera) -> Tuple{Vararg{Symbol}}The fixed that holds every parameter of camera in a refine (refine_camera_and_poses): a parametric camera's parameter names (parameter_names), a SplineLens camera's (:controls,), its route's one parameter group. A refine that requires a held camera, the rotation-centre views', takes it as fixed = held_parameters(camera).
camera is a CameraModel or a BoundedCamera of one.
VisualGeometryToolkitEstimation.RotationCenterRefineMetadata — Type
RotationCenterRefineMetadataThe metadata of refine_camera_and_poses on RotationCenterViews: the bundle's robust record, as CameraPosesRefineMetadata's and in its units, PIXELS — each dot's bearing row is its tangent residual whitened into pixels by the back-projection Jacobian, so residuals (px), s² (px²), ν, residual_norms (one per correspondence, px), weights, scale (px) and loss_width (px), with scale_source — and the dots without a bearing (the camera does not back-project them, or their back-projection Jacobian cannot whiten the residual), which make no row: read by n_without_bearing and without_bearing (their (view, correspondence index) pairs).
VisualGeometryToolkitEstimation.n_without_bearing — Function
n_without_bearing(r) -> IntThe number of dots a rotation-centre refine made no row of (no bearing): of its result r or its RotationCenterRefineMetadata.
VisualGeometryToolkitEstimation.without_bearing — Function
without_bearing(r) -> AbstractVector{Tuple{K, Int}}The dots a rotation-centre refine made no row of, as (view, correspondence index) pairs (the view keyed as the input): of its result r or its RotationCenterRefineMetadata.
VisualGeometryToolkitEstimation.without_projection — Function
without_projection(r) -> AbstractVector{Tuple{K, Int}}The dots a rotation-centre refine's ReprojectionResidual stage made no row of — they back-project, but do not project at the bearing estimate — as (view, correspondence index) pairs: of its result r or its RotationCenterRefineMetadata. Empty for the bearing residual.
VisualGeometryToolkitEstimation.AbstractViewResidual — Type
AbstractViewResidualThe residual of refine_camera_and_poses on RotationCenterViews, its fourth argument:
BearingResidual()(the default): each dot back-projected ONCE to its bearing, the tangent-plane angle to the predicted bearing whitened into pixels by the dot's back-projection Jacobian — never a projection inside the solve, so it converges from a rotation-centre start whatever the lens's domain;ReprojectionResidual(): that bearing refine, then a SECOND stage from its estimate on the exact pixel reprojection error,project(camera, R_k (X − c) + d) − u(Core'sreprojection_error; a SplineLens camera projected by inverting its backward spline, its forward dropped, Core's implicit-function derivatives). A candidate update that leaves a dot's prediction undefined (off the lens's domain) costs +Inf and is rejected by the LM, never clamped. The dots of the second stage are the first stage's rows that project at its estimate; those that do not arewithout_projection. The result's views, retcode, rms and record are the second stage's.
VisualGeometryToolkitEstimation.BearingResidual — Type
BearingResidual()The default AbstractViewResidual of refine_camera_and_poses on RotationCenterViews: the bearing refine alone.
VisualGeometryToolkitEstimation.ReprojectionResidual — Type
ReprojectionResidual()The AbstractViewResidual of refine_camera_and_poses on RotationCenterViews that adds the second stage on the exact pixel reprojection error; the dots that do not project at the bearing estimate are without_projection.
Every bundle — this one, refine_robot_world_hand_eye and a downstream package's (PanoramaStitching's rotation bundle) — reads its robust loss from its keywords through one function:
VisualGeometryToolkitEstimation.bundle_loss — Function
bundle_loss(optimization_strategy; scale | width, codimension = 2, caller = "bundle_loss")
-> (; strategy, input, loss)The robust loss a bundle adjustment deploys, from its keywords, by the refine contract every refine_* follows: optimization_strategy a Triggs{K} (a bundle's default, its TOML table's: the one fixed-scale robust path, one robustified LM) or nothing (plain least squares), refused otherwise — an Irls{K} naming its Triggs{K}, a GaussianShape strategy (least squares is nothing) and a strategy holding an absolute loss_width (that is the width keyword); and EXACTLY ONE of scale = σ (the observations' noise scale, or an upstream result; VisualGeometryToolkitCore's scale_input) or width = w for a robust strategy, NEITHER for least squares (a bundle refine estimates no scale). The loss is the strategy's shape deployed at the residual's codimension (2 for a pixel reprojection error): at c × scale, c the strategy's efficiency constant (the one it states, the plain multiple, or Core's efficiency_constant(K, codimension); a truncation states one), or at width — which refuses a strategy that also states a constant. caller names the bundle in the messages (refine_camera_and_poses: …).
Returns strategy (the strategy, or nothing), input (the ScaleInput, or nothing) and loss (VisualGeometryToolkitCore's DeployedLoss, or nothing): a bundle robustifies its rows with loss_robustifier(strategy_loss(strategy)(), sqrt(sq_loss_width(loss))) and records the scale it was given through the stack's accessors: scale_source(input) (ScaleNone for least squares, input === nothing, and for a width) and, when that is not ScaleNone, noise_scale(input). Shared by refine_camera_and_poses, refine_robot_world_hand_eye and downstream bundles (PanoramaStitching's rotation bundle), so one keyword set means one loss everywhere.
Penalty weights: normalization and generalized cross-validation
One convention for a penalty's weight across the stack: λ = β · energy / N / m̄ (VisualGeometryToolkitCore's penalty_scale), β relative to the data's energy per parameter and to the penalty operator's mean positive eigenvalue m̄ (Core's mean_positive_eigenvalue), so β = 1 weighs the penalty like the data. gcv_select chooses β by generalized cross-validation from a score of λ — a coarse scan of log₁₀ β, then Brent's method — the score the closed-form curve of one penalized least squares (gcv_curve: every λ from one generalized eigendecomposition). A nonlinear fit evaluates it on its linearization (one solve at β = 1, the curve, one final solve). Public, not exported.
VisualGeometryToolkitEstimation.gcv_curve — Function
gcv_curve(G, h, rr, Q, q, n) -> (λ -> (; gcv, rss, dof, y))GCV of the penalized least squares min_y ‖J y − r‖² + λ‖D y − c‖² with the normal-equation pieces G = JᵀJ, h = Jᵀr, rr = ‖r‖², Q = DᵀD, q = Dᵀc and n data rows, as a function of λ. One Cholesky B = G + Q = L Lᵀ and one symmetric eigendecomposition L⁻¹ Q L⁻ᵀ = U diag(μ) Uᵀ — the generalized eigenproblem of the pencil (Q, B), V = L⁻ᵀ U: VᵀBV = I, VᵀQV = diag μ — diagonalize every M_λ = G + λQ at once, VᵀM_λV = diag s, sᵢ = 1 − μᵢ + λμᵢ. With zh = Vᵀh, zq = Vᵀq and w = (zh + λ zq) ./ s (the solution y = V w, returned: a nonlinear fit's Gauss–Newton step at the chosen λ): rss = rr − 2 w⋅zh + Σ (1 − μᵢ) wᵢ², dof = tr H_λ = Σ (1 − μᵢ)/sᵢ. Parameters nothing reads (zero diagonal in B) are dropped (0 in y). B must then be positive definite: a direction no row holds is a gauge, which its caller holds exactly (Core's rule: no penalty or ridge fixes it); an ArgumentError otherwise. Dense, O(p³) once in the parameters, then O(p²) per λ. (Moved from RadiometricCamera's linear solves.)
gcv_curve(J, r, D, c; exact_max = 3000, probes = 64, seed) -> (λ -> (; gcv, rss, dof, y))The same from the rows themselves (J, D dense or sparse): the pencil above for at most exact_max parameters, the data residual r − J y read from the rows (exact where it is small: the normal-equation form's rr − 2w⋅zh + … cancels); past it, per λ one sparse Cholesky of M_λ (CHOLMOD, its symbolic analysis done once: the pattern does not depend on λ) for the solution and a Hutchinson estimate of tr H_λ = tr(M_λ⁻¹JᵀJ) from probes Rademacher vectors drawn from seed (deterministic; the probes read the symmetric zᵀJM_λ⁻¹Jᵀz, H's eigenvalues in [0, 1], so its standard deviation is ≤ √(2 tr H / probes)). Either way the parameters nothing reads are dropped (0 in y).
VisualGeometryToolkitEstimation.gcv_select — Function
gcv_select(score, scale; log10_bracket = (-8.0, 4.0), points = 25)
-> (; β, λ, gcv, rss, dof, log10_β_bracket, at_bracket_end)The penalty weight β minimizing GCV, score(λ) returning (; gcv, rss, dof) at λ = β · scale (Core's penalty_scale) — a gcv_curve, or a caller's own evaluation: a coarse scan of points values of log₁₀ β, evenly spaced over log10_bracket and evaluated from the largest down, then Brent's method on the scan's interval around its best point, to Brent's own floor (Core's grid_brent_minimum; no tolerance to choose: each evaluation of a curve is closed form). GCV need not be unimodal over many decades: the scan picks the basin. A least value at a scan end is refined on its one neighbouring interval, and at_bracket_end says the result lies at that end (the bracket did not contain the minimum; never silently taken as interior). gcv is the least value evaluated, dof the effective number of parameters tr H_λ there. A score of Inf (a weight the caller could not evaluate) is never the minimum. (RadiometricCamera's scan-and-Brent search, adopted.)
VisualGeometryToolkitEstimation.gcv_score — Function
gcv_score(n, rss, dof) -> Float64The generalized cross-validation score n·rss / (n − dof)² of a fit with n data rows, residual sum of squares rss and dof = tr H effective parameters.
VisualGeometryToolkitEstimation.check_penalty_weight — Function
check_penalty_weight(w, what) -> wA penalty weight as an option states it: a finite β ≥ 0 (no penalty is β = 0), or :gcv (chosen by gcv_select). An ArgumentError naming what otherwise. The keyword allow_nothing (which also admitted nothing) is retired (0.11.0: no caller left), refused by name.
Camera-model regression
Three verbs regress one camera model onto a source camera over a region of its sensor, by the stack's naming rule (a fit has no guess, a refine needs one):
fit_camera(T, source, region; …)is LINEAR: one least-squares solve. For a lens familyLit solves the focal rescale and the family's coefficients from the samples' radial relation where the family has a linear form (a lift:Division,Scaramuzza; a fixed projection with an even radial factor: Kannala–Brandt, the polynomial and rational OpenCV models, Fisheye624), and refuses a family with none (DoubleSphere,Unified,ExtendedUnified, aTiltorFieldOfViewterm) with anArgumentErrornaming the two verbs below. Its targetsSplineLens(aCameraModelwhose lens is the backward spline of rays over the image) andNoncentralSplineModel(its direction and point splines) are penalized linear least squares; a SplineLens SOURCE is instead resampled exactly (Core'sextend_latticeover the region,subdivide_latticeto thelattice_spacingcell, then the turn onto its optical axis), its extension and fold report recorded;region ⊆ valid_domainof the result always holds, a region on the source's lattice edge included (each edge exact where a float lands on it, else one step of K′'s map outward).pspline_grid(T, source, region, grid_size)is the spline targets' one penalized solve on a given lattice,grid_size = (columns, rows)in Core's order;fit_camerarepeats it at each spacing it tries.refine_camera(guess, source, region; …)is the chart LM from a guess to the exact pixel least-squares minimum of the guess's family (for a SplineLens source also its principal point, aspect and axis).approximate_camera(L, source, region; …)is the two:refine_camera(estimate(fit_camera(L, source, region)), source, region), a family with no linear form starting from its anchor. A bounded source is fitted overregion ∩ imaged_region(source): no sample where the camera images nothing.
A SplineLens camera's K is a gauge, so its ẑ and principal point are wherever its frame put them. optical_axis(camera::BoundedCamera[, region]) finds the lens's optical axis: the closest radially symmetric camera over the camera's own bound (a circular fisheye's image circle keeps its corners out), or over a given image region within it (a target's rectangle: the pixels whose rays are measured), with its principal point and its axis free. It fits as approximate_camera does, order by order (1 → 2 → 3 → 4 radial terms, each refine from the last order's camera), and keeps the highest order that holds by its own validity gate — converged, its regression stationary at the estimate (on the axis), and not folding inside the region (metadata(r).family.fitted_terms). fit_camera(SplineLens, …) and the SplineLens refine_camera_and_poses turn the camera onto that axis first (Core's axis_gauged).
fit_forward_spline(camera; tolerance) gives a SplineLens camera its forward spline and keeps everything else — its backward, K and frame — so poses estimated against it stay valid; with a forward, project evaluates the spline and runs no projection solve. It runs the stack's one forward fit (the fit fit_camera(SplineLens, …; forward = true) and the SplineLens refine_camera_and_poses' refit run).
VisualGeometryToolkitEstimation.fit_camera — Function
fit_camera(::Type{T}, source, region; …) -> FitResult{<model of T>, CameraFitMetadata}The camera model of type T regressed onto source over region, a region of the source's sensor in its pixel frame (an imaged_region, an Ellipse, a Rect, their ∩; bounded, and inside the source's calibrated support — the keyword support, its valid_domain by default, or a re-gridded camera's calibration's — else Core's RegionBeyondSupport at entry; for a SplineLens source, whose lattice the resample extends over the region, only a support the caller states), by ONE linear least squares — a fit has no guess. The objective is the integrated squared pixel error of the round trip, ∫ ‖π_T(π⁻¹_source(u)) − u‖² du over the region. The target type selects the family and the solve:
- a lens family
L <: AbstractLensModel(fit_camera(L, source::CameraModel, region; weight = :image, fixed = ())): aCameraModelwith lensL, its focal rescales(fx′ = s·fx₀, the skew with it) and its coefficients from the samples' radial relation — radiusρagainst view angleθ— in one solve. The family's class decides the linear system:- a lift (
LiftLens:Division,Scaramuzza, BabelCalib's rational lift): the lifted ray(x·N(ρ), D(ρ))is parallel to the sample's ray, linear in every coefficient and the rescale, its rows weighted to the angle residual times the radius to first order; - a fixed projection
g(θ)(Perspective,Equidistant,Stereographic,Equisolid,Orthographic) with an even radial factorN(g)/D(g)and decentering deltas — Kannala–Brandt, the polynomial and the rational OpenCV models, Fisheye624:ρ·D(g) = s·g·N(g), linear ins,s·kandd. Without a denominator the row's residual is the pixel error itself, so the fit IS the integral's least squares at a physicalK's samples; with one (the OpenCV rational model) the rows are divided by the denominator of a first solve, once. The decentering deltas are zero (the radial relation does not see them); - no linear form: a projection with a shape parameter (
DoubleSphere,Unified,ExtendedUnified), aTiltorFieldOfViewterm — anArgumentErrornamingrefine_cameraandapproximate_camera.
K, by Core'sIntrinsicsRole:- A PHYSICAL
K(a parametric lens): the principal point and ẑ are kept. The samples are the source lens's radial profiler(θ)(its radial part: decentering terms are stripped, warned when non-negligible) on Clenshaw–Curtis nodes. The integral over the region reduces exactly to one overθ, with the density weighted by the fraction of each radius's circle about the principal point that lies in the region (the region must be star-shaped about the principal point, as a convex one containing it is). The domain is the region's farthest radius. A region reaching past the radii the source images reaches past itsvalid_domainand is refused at entry (Core'sRegionBeyondSupport), never clipped (0.11.0; it was clipped with a warning). A region reaching past the TARGET family's view-angle limit (Core'sview_angle_limit: a gnomonic family applies its distortion on the perspective plane, which ends at 90°) is aViewAngleLimitError(the region's angle and the limit in its fields): the family cannot represent the source there, and it is never fitted over a clipped field. (The region's extent is measured along 720 azimuths: a corner between two of them is reached to within half an azimuth's arc.)weightis the density::image(uniform on the image),:solid_angle(sin θ) or:uniform(inθ). - A GAUGE
K(aSplineLenscamera source, aCameraModelor aBoundedCameraof one) has no radial profile of its own lens. The samples are the CAMERA's rays on the image over the region (the spline targets' quadrature, below, back-projected on the backenddevice,default_device()by default), each pixel one sample of equal weight: the same image integral with the:imagedensity (anotherweightis anArgumentError). The linear fit reads their radii about the source's principal point and their angles about its ẑ; the principal point, the aspect and the axis are freed byrefine_camera(optical_axis). This is the route to a parametric lens, an OpenCV or COLMAP form, for a SplineLens camera:approximate_camera(KannalaBrandt{4}, camera, region). - The result reports the linear fit's residual: the round trip's error of the returned camera at each sample, in pixels (
metadata.residuals,max_px,max_px_edge; the result'srmstheir weighted root mean square). Theretcode_symbol(result)is:solved,iterations0; a degenerate solve (a non-positive or non-finite rescale) is:solve_failed(its camera's focals non-finite), never a throw. fixed(fixed = (): every coefficient and the focal solved) holds parameters by name or group, resolved by Core'sfree_parameter_namesasrefine_camera's: a held coefficient's column leaves the ONE linear solve and keeps the target's value — the family's anchor (zero) for a typeL, or a given LENS's,fit_camera(target::AbstractLensModel, source, region; fixed)(the target lens's family, its held values its own: a calibration's lens target);:fheld is the source's focal (s = 1);fixedcovering everything returns the target unchanged. The decentering deltas the radial relation does not see keep the target's values. A lift's linear form rescales every coefficient with the focal, so a lift holding a coefficient must hold:ftoo (anArgumentErrorotherwise). The keywordsfree,fix,fixed_paramsandholdare refused, namingfixed.Divisionalone fitsDivision{1}; any other type must carry its arities (KannalaBrandt{4}). A non-central source (NoncentralSplineModel) is anArgumentError: not supported yet.
- a lift (
SplineLens(fit_camera(SplineLens, source, region; tolerance, grid_size = nothing, lattice_spacing = nothing, forward = true, device = default_device())): aCameraModelwith a VisualGeometryToolkitCoreSplineLens, itsKthe source's own (skew 0). Its BACKWARD spline is Core's full-image lattice on the source's image (spline_lattice; the source is not sampled: the solve sets every control), fitted to the source's rays at the normalized pointsx = K⁻¹uby linear least squares with the P-spline penalty —pspline_gridatgrid_size(columns, rows; reported back in the metadata in that order, as Core'sgrid_size(camera)) when given, else at the lattice THE SPACING SEARCH chooses: the spacing is doubled down from 4 cells along the longer side and then bisected to the LARGEST spacing whose held-out maximum error is at mosttolerance(px),retcode_symbol:converged; a fit that cannot reach it at the finest spacing of the bracket (a cell of three quadrature strides) returnsretcode_symbol:max_iterationswith what it reached. The region restricts the quadrature only, never the lattice: a node whose support misses it is held by the penalty. The tolerance bounds the held-out DIRECTION error in pixels, the non-central target's metric (below). A physical source's lattice takes its own K and ẑ, already the axis. A SPLINELENS SOURCE is resampled EXACTLY, never refitted (lattice_spacing = nothing, px, a number or(sx, sy)): its lattice extended on its own spacing until every node whose support meetsregionexists (Core'sextend_lattice: its nodes bit for bit, the new ones by Core's bending-energy continuation, marked), subdivided by the integers whose cell is nearestlattice_spacing(near ties toward the target's cell aspect, square for one number;nothingkeeps the source's spacing) by Core's exact knot insertion (subdivide_lattice), and turned onto the source's optical axis:optical_axisof the source over its own bound (anapproximate_camera; refused when it does not converge), Core's exactaxis_gauged— K's principal point at the axis's pixel, every ray turned (the metadata'srotation, itssymmetry_residual). Inside the source's domain the backward is the source's function to rounding; its held-out error against the turned source's rays (max_px,max_px_edge,rms, where the source images a ray) states it.regionmay exceed the source's valid domain (noRegionBeyondSupport), andregion ⊆ valid_domainof the result holds (Core's own⊆), a region on the source's lattice edge included: each edge of that domain is the extended lattice's, or the region's where the region reaches past it within the extension's rounding tolerance (1e-9 of a cell, so no ring is added), exact wherever a float lands on it, otherwise rounded OUTWARD to the nearest edge Core's map of the lattice's area through K′ reaches: one step of that map (fx·eps(x) px; 2.8e-14 px for a 640 × 480 source), so the domain claims at most that step beyond the data. Core's⊆and the gauge K′ are unchanged. Agrid_sizewith a SplineLens source is anArgumentErrornaminglattice_spacing, and alattice_spacingwith any other source is one too. Its FORWARD spline (forward = true) is aChartSpline{2}from the equidistant direction chart about ẑ tox, fitted to the camera's own rays over the region by the same penalized solve and the same spacing search, totolerancein pixels of the held-out round trip; the camera projects through it (NaN outside the fitted chart). Each fit of a search is an outer iteration, counted in the metadata;iterations(result)is 0; the retcode is:convergedwhen every search met the tolerance.NoncentralSplineModel(fit_camera(NoncentralSplineModel, source, region; tolerance, grid_size = nothing, device = default_device())): the same lattice and search on the source's observation LINES — a non-central model'sRay3s, or a central source's rays through its centre. Both splines, the unit directions and the points, come from ONE penalized solve on the same lattice. A line's point is pinned by an anchor: c, Core'snearest_pointof all the quadrature lines, and each line contributes its point closest to c,o + ((c* − o)·d) d— bounded for rays at and past 90°, where a plane anchor (the pupil plane) runs off. The points are in the source's frame (frame in = frame out; plain metres, the model's unit), c* in the metadata; a central source's are all c*. The tolerance bounds the held-out DIRECTION error in pixels: the angle to the source's ray scaled by the lattice K's focal length (f·angle, where the source images a ray); the line offset is reported beside it, in metres.
The spline targets' source is a BoundedCamera — the lattice is laid over its image (a bare model has none: an ArgumentError) — of a CameraModel, and for the non-central target also of a NoncentralSplineModel. Their quadrature is on the image (they take no weight). Their per-point steps run as batches: a SplineLens source's backprojection is Core's KernelAbstractions kernel on the backend device (default_device() by default: an accelerator when one is functional), its rays brought back to the host; any other source's backprojection, the chart, the design rows, the evaluation and the errors run on the host (pointwise!), and the penalized normal equations are solved by a sparse Cholesky factorization.
The metadata (CameraFitMetadata) records the region, the round trip's residuals, max_px and max_px_edge, the family's own record and no scale (ScaleNone: no camera regression has a robust loss); the result's rms is the round trip's root mean square in pixels. For the spline targets these are their held-out errors.
region = imaged_region(with_lens_coverage(camera, image_circle(camera, 8.0mm)))
kb = fit_camera(KannalaBrandt{4}, model(camera), region) # a parametric lens, linearly
kb = approximate_camera(KannalaBrandt{4}, model(camera), region) # and its exact pixel least squares
sl = fit_camera(SplineLens, camera, region; tolerance = 1.0e-3) # its backward grid and forward spline
project(estimate(sl), ray) # its pixel through the forward, NaN outside the fitted chartVisualGeometryToolkitEstimation.fit_forward_spline — Function
fit_forward_spline(camera::BoundedCamera; tolerance, region = imaged_region(camera), device = default_device())
-> FitResult{<:BoundedCamera}Give a SplineLens camera its FORWARD spline, the camera itself kept: its K, its backward lattice (the camera's rays), its presence mask and its frame exactly as they are, any forward it held replaced. The forward is a ChartSpline{2} from the equidistant chart about ẑ (Core's forward_chart(SplineLens)) to the normalized point, fitted to the camera's own rays over region (pixels) with the P-spline penalty, its knot spacing searched until the held-out round trip ‖K·forward(chart(ray)) − u‖ meets tolerance (pixels) — the one forward fit of the stack, fit_camera(SplineLens, …; forward = true)'s. With a forward, project evaluates it inside its fitted chart: no projection solve, no cold start.
Unlike fit_camera(SplineLens, camera, region), which turns a gauge-K camera onto its optical axis first (a new frame), this keeps the frame, so poses estimated against the camera stay valid.
The result's estimate is the bounded camera with its forward; its rms and the metadata's max_px / max_px_edge are the held-out round trip's (pixels), with the knot spacing, the counts of controls, λ and the search's outer_iterations. Any other camera (a lens that is not a SplineLens, a bare model) is an ArgumentError.
VisualGeometryToolkitEstimation.refine_camera — Function
refine_camera(guess::CameraModel, source, region; fixed = (), weight = :image, max_iter, device = default_device(),
optimization_strategy = nothing, support = valid_domain(source)) -> FitResult{<:CameraModel, CameraFitMetadata}The camera of guess's lens family polished onto source over region (a region of the source's sensor in its pixel frame, as fit_camera takes it): the exact minimizer of the integrated squared pixel error of the round trip, ∫ ‖π(π⁻¹_source(u)) − u‖² du, by the stack's chart LM from guess. It is a refine: it needs the guess, and it converges to the minimum nearest it. fit_camera gives a guess where the family has a linear form; approximate_camera is the two in one call.
The region's support. The region must lie inside the source's calibrated support — support, the source's valid_domain by default, or the support of the camera it was re-gridded from — or the fit is refused with Core's RegionBeyondSupport (check_region_support, at entry): a fit never reads a model where it is extrapolated.
- What moves: everything but
fixed, the guess's parameters held by name (parameter_names(guess): the intrinsics':f,:aspect,:cx,:cy, then the lens's) or by group (a coefficient family's stem::kthe radial numerator,:dits denominator,:pthe tangential terms,:sx/:sythe thin prism's); the gauge's skew is held by construction. A held parameter keeps the guess's value.:fis the focal rescalesfrom the guess's focal (relative to the source's,fx′ = s·fx₀); a forward lens's radial factor moves its numerator and its denominator (fixed = (:d,)holds the denominator); a lift moves every coefficient. A name or group the guess does not have is anArgumentError, and so are the keywordsfree,fix,fixed_paramsandhold(namingfixed). - A physical
K(a parametric source): the principal point, the aspect and ẑ are the source's, data, so the guess's principal point must be the source's (anArgumentErrorotherwise) and:aspect,:cx,:cyare held, as are the decentering terms (tangential, thin prism) its azimuth-0 profile does not see. The samples are the source lens's radial profile (its radial part: decentering terms are stripped, warned when non-negligible) on Clenshaw–Curtis nodes, the exact reduction of the image integral toθ, with the densityweight(:image,:solid_angleor:uniform) and the region's coverage of each radius (the region star-shaped about the principal point). - A gauge
K(a SplineLens camera, aCameraModelor aBoundedCameraof one): the principal point, the aspect and the axis are free, from the guess's principal point and aspect and from the source's ẑ. The samples are the camera's own rays on the image over the region, each pixel one sample of equal weight (the:imagedensity; anotherweightis anArgumentError). The fitted camera's principal point is the axis's pixel, andmetadata.rotationis the turn from the source's frame to the fitted one;metadata.symmetry_residual(px) is the azimuthal misfit no radially symmetric camera represents.optical_axisis this refine of the Fisheye624 form (4 radial terms). - The residual is one 2-D residual per sample: a forward lens's is its pixel minus the sample's (
projected_radiusalong the ray's azimuth, finite past a fold of its profile); a lift's is angle-style, in the equidistant chart about ẑ, scaled by the sample's normalized radius. - The device. Each sample's residual and its Jacobian rows, and their reduction to the normal equations, are one KernelAbstractions kernel on the backend
device; only the chart LM's N × N step is the host's. - The loss.
optimization_strategy = nothing, the plain least squares, is the only one: a camera-from-camera regression has no outliers. It takes noscaleorwidth(one given would be ignored: anArgumentError), and the result records no scale (scale_sourceScaleNone,scaleNaN;noise_scalerefuses it).
The domain is the region's farthest radius, capped by the source's own view-angle limit, as fit_camera's; a region past the family's view-angle limit (the perspective plane's 90° for a gnomonic family) is a ViewAngleLimitError. At most max_iter accepted iterations (configs/default.toml [refine_camera]); the termination record (retcode, stop_rule) is the LM's, a refine that ends at a non-finite focal rescale is (ReturnCode.Failure, NonFinite) (:solve_failed), never a throw, and one with nothing free (the guess returned, no LM) is a closed form, (ReturnCode.Success, ClosedForm). The metadata is CameraFitMetadata: the round trip's errors per sample in residuals (px), their maximum and edge-band maximum, the result's rms their weighted root mean square, and family.stationary: whether the regression is stationary at the estimate (Core's chart-LM postcondition; Core 0.11.0 reads a tolerance stop Success without it, so a converged refine can stop short of the minimum).
A NoncentralSplineModel source is an ArgumentError: not supported yet.
start = estimate(fit_camera(KannalaBrandt{4}, camera, region)) # the linear fit: the guess
kb = refine_camera(start, camera, region) # its exact pixel least squares
kb = approximate_camera(KannalaBrandt{4}, camera, region) # the same two stepsVisualGeometryToolkitEstimation.approximate_camera — Function
approximate_camera(::Type{L}, source, region; weight = :image, max_iter, device = default_device(), support = valid_domain(source))
-> FitResult{<:CameraModel, CameraFitMetadata}The camera of lens family L closest to source over region (a bounded source: region ∩ imaged_region(source), no sample where it images nothing), from no guess: refine_camera(estimate(fit_camera(L, source, region)), source, region) — the linear fit's camera is the refine's start (fit_camera, refine_camera). The keywords split to their stages: weight and fixed to both (the held coefficients keep the family's anchor values through both), device to both (a gauge-K source's rays are sampled on it), max_iter to the refine. The result is the refine's.
A family with no linear form (a projection with a shape parameter — DoubleSphere, Unified, ExtendedUnified — or a Tilt or FieldOfView term) starts from its anchor instead: the family's zero lens (the double sphere at ξ = α = ½) at the source's K. So does a linear fit that failed (:solve_failed). Division alone names Division{1}.
kb = approximate_camera(KannalaBrandt{4}, camera, region) # a SplineLens camera's parametric lens (the exporters' route)VisualGeometryToolkitEstimation.optical_axis — Function
optical_axis(camera::BoundedCamera, region = imaged_region(camera); support = valid_domain(camera), device = default_device())
-> FitResult{<:CameraModel, CameraFitMetadata}The radially symmetric camera closest to camera over region ∩ imaged_region(camera): by default the camera's own bound, its imaged_region (the sensor ∩ its lens coverage ∩ the model's valid domain); given a region (a Rect2, an Ellipse, or their ∩, in the camera's pixel frame), its part the camera images — a target's rectangle, where the rays are measured, keeps out a lattice's unconstrained edge (RadiometricCamera's siemens start camera stalls over its whole image and converges over its board). It approximates camera by the Fisheye624 form (COLMAP's RAD_TAN_THIN_PRISM_FISHEYE, Lens(Equidistant(), Additive(Tangential, ThinPrism{2}) ∘ Radial{Even, n}), with n = 4 radial terms, θ³…θ⁹: configs/default.toml [optical_axis] radial_terms, the smallest order that locates the axis as well as 6 terms do on the cameras studied, and far better conditioned; its decentering held at zero), as approximate_camera does — the linear fit of its radial coefficients and focal, then the refine with the principal point and the axis free — but order by order (below): on a source the 4-term start fits well, it lands where approximate_camera of that family does.
- The region is the bound or a part of it, never more. A circular fisheye's lens coverage (its image circle) keeps the corners out, where a grid's rays are its extrapolation. A whole-sensor fit is a camera whose bound is the whole sensor.
- A bare model has no bound: an
ArgumentError. Wrap it inBoundedCamera(model, size; lens_coverage). - For a SplineLens camera (a gauge
K), the principal point and the axis are free. Read them from the result.- The optical axis in the camera's frame is
rotation(metadata(r).rotation)' * ẑ. - Its pixel is the fitted camera's principal point.
metadata(r).symmetry_residual(px) is the azimuthal misfit that no radially symmetric camera represents. A decentred lens reports it.
- The optical axis in the camera's frame is
- A physical
Kis its own axis: the identity rotation and the source's principal point.
device is the backend the camera's rays are sampled and the regression reduced on (approximate_camera's). support bounds the region (region must lie inside it, else Core's RegionBeyondSupport; never clipped): the camera's valid_domain by default; for a camera re-gridded from a calibration (a whole-sensor lattice extrapolated past the board), the calibration's support — RadiometricCamera's siemens start camera, refused over its whole sensor, fits over its Schöps support.
The fit is an ORDER CONTINUATION. The family is fitted at [optical_axis] start_radial_terms radial terms (1: its linear fit, then the refine), then at each next order up to radial_terms (2, 3, 4), each order's refine starting from the last order's camera, its higher coefficients held at 0. A low order does not follow a source's profile where it bends at the region's edge, so its start never folds there; raising the order one term at a time keeps each refine in the basin of the last (a direct lift from 1 to 4 terms can end in a worse local minimum). An order HOLDS when its refine converged, its regression is stationary at the estimate (family.stationary: on the axis, Core 0.11.0 reading a tolerance stop converged without it) and its lens does not fold inside the region (view_angle_limit below θ_max); every order is refined, each from the last one's camera whether or not it held (a rectilinear lens turning at the rim: its 1- and 2-term fits fold inside the region with the axis unmoved, its 3-term one does not), and the result is the highest order that held, metadata(r).family.fitted_terms (its higher coefficients 0).
When no order holds, the fallback is the whole-order linear start. A start that FOLDS before the region's widest angle (a rectilinear lens to 62° whose 4-term fit turns over at 56°) is recorded as metadata(r).family.start_fold (radians; nothing when the start does not fold); when the refine from it does not hold (the same gate: converged, stationary, no fold inside the region), the axis is fitted again over the part of the region inside the fold (out to 0.9 of the fold angle, about the start's principal point), metadata(r).region then the region used; a start folding just past the widest angle, inside the band that refit leaves out, is refitted the same way. An axis fit that still fails the gate (from any start, folding or not) is an ArgumentError: the axis re-gauge is applied always (fit_camera(SplineLens, …)), and a stalled fit is never a silent rotation 0.
Core's axis_gauged(camera, rotation(metadata(r).rotation), calibration_matrix(estimate(r))) then turns the camera so ẑ is the axis. fit_camera(SplineLens, …) does this first, always (a held SplineLens camera's refine_camera_and_poses does not: it returns the camera as given).
VisualGeometryToolkitEstimation.pspline_grid — Function
pspline_grid(T, source, region, grid_size; device = default_device(), support = valid_domain(source)) -> FitResult{<:model of T, CameraFitMetadata}The P-spline (penalized B-spline least squares, Eilers–Marx) of a spline camera's lattice onto source over region, on the GIVEN lattice: T is SplineLens (a CameraModel with a SplineLens, its backward spline, no forward) or NoncentralSplineModel (its direction and point splines). The lattice is Core's full-image layout of grid_size = (columns, rows) nodes on the source's image (spline_lattice, the source's own K), the region restricting the quadrature only, ONE penalized solve with the third-difference penalty fit_camera uses — the non-central model's points anchored on c*, its directions and points one right-hand side — in the source's frame. No spacing search and no tolerance: the one deterministic least-squares step fit_camera(T, source, region; tolerance) repeats at each spacing it tries (the same implementation), so pspline_grid(T, source, region, (gw, gh)) at the lattice fit_camera chose is its backward — for a gauge-K source (a SplineLens camera) once that source is turned onto its optical axis, the frame fit_camera fits in.
source is a BoundedCamera (the lattice needs the image: a bare model is an ArgumentError) of a CameraModel or, for the non-central target, also of a NoncentralSplineModel. The result's retcode_symbol is :solved, iterations 0; its metadata is fit_camera's, the held-out errors (the direction error in pixels; for the non-central model also the line offsets in metres, and center) and family = (; λ, grid_size, stride, n_held_out_dropped, …) with grid_size the lattice's (columns, rows), as fit_camera reports it, and no tolerance or outer_iterations.
VisualGeometryToolkitEstimation.CameraFitMetadata — Type
CameraFitMetadata{R, F}FitResult metadata of fit_camera, refine_camera and approximate_camera: the region fitted over, the weight measure (:image for every target but a lens family), the round trip's error in pixels — residuals (per sample), max_px (their maximum) and max_px_edge (their maximum within 100 px of the region's edge); √ of their weighted mean square, the objective's quantity, is the result's rms — the target family's own record family, a NamedTuple, and:
rotation: from the source's camera frame to the fitted camera's, a frame-labelledRotationMap(ViewFrame{:source}→ViewFrame{:fitted}; one type for every fit). For a source whoseKis physical it is exactly the identity: its ẑ is the axis, not fitted. A gauge-K(SplineLens) source's refine gives the fitted rotation, for a lens family and for the SplineLens target (its axis gauge) alike; a linear fit keeps the source's ẑ.symmetry_residual: px. The root mean square of the round trip's azimuthal component about the fitted principal point, the misfit that no radially symmetric camera about the fitted axis represents. It isNaNwhere not measured: a physicalK's radial profile, a lattice fit of a physical source.weights,scale,scale_source: the robust stages' weight and scale record. No camera regression has a robust loss: its weights are Core's lazyunit_weights(one per sample, binary byweight_kind),scaleisNaNandscale_sourceScaleNone, sonoise_scale(result)refuses to hand it on.
The family records:
- a lens family:
(; focal_scale, aspect, θ_max, θ_source). These are the joint focal rescales, the aspectfy′/fy(1 for a physicalKand for a linear fit) and the fitted and the source's angular domains (radians); a region past the radii the source images is refused at entry, never clipped. The errors are at the quadrature's samples; SplineLens:(; λ, grid_size, stride, outer_iterations, n_held_out_dropped, extrapolated_fraction, tolerance, forward),forwardbeing(; spacing, counts, λ, outer_iterations, max_px, max_px_edge, rms)ornothing— the backward lattice's P-spline weightλ(nothingfor a resampled SplineLens source), its(columns, rows)(Core'sgrid_size), the quadrature's stride in pixels, the spacing search's fits (0 for a given or resampled lattice), the held-out pixels dropped (below), the share where the backward has no source ray (extrapolated_fraction: of the image's area, the full-image lattice's, for a fitted lattice — outside the region, or where the source images none, held by the penalty alone; of the region's pixels for a resampled SplineLens source — 0 inside its domain), and the tolerance asked for. A resampled SplineLens source addssubdivision(the integers(m_x, m_y)),lattice_spacing(the resulting cell, px),extension_offset(Core'sextend_latticeoffset: the source's first node in the extended lattice),extrapolated(the resulting lattice's nodes Core continued, aBitMatrix[row, column]) andfolds(Core'sfolded_pixelsreport over the region, as given; reported, never acted on). Then the forward spline's knot spacingh(radians in the chart), its control counts per chart axis, itsλ, its search's fits and its held-out round trip‖K·forward(chart(ray)) − u‖in pixels (maximum, edge-band maximum, rms) —forwardisnothingwithforward = false. The result's errors (residuals,max_px,max_px_edge,rms) are the backward's held-out direction error, as the non-central target's (below): the angle to the source's ray,f·anglein pixels, over the held-out pixels where the source images a ray (n_held_out_droppedis 0: no pixel is dropped from them since 0.11.6);NoncentralSplineModel:(; λ, grid_size, stride, tolerance, outer_iterations, center, max_offset, rms_offset, max_offset_edge, n_held_out_dropped)—centerthe anchor c* (the least-squares point nearest to the source's quadrature lines, in the source's frame, plain metres as the model's points), and the held-out line offset‖(I − ddᵀ)(o_fit − o_src)‖(plain metres); the pixel errors are the held-out DIRECTION error, the angle to the source's ray scaled by the lattice K's focal length (f·angle).
VisualGeometryToolkitEstimation.ViewAngleLimitError — Type
ViewAngleLimitError <: ExceptionThe refusal of a camera-from-camera regression (fit_camera, refine_camera, approximate_camera) whose region reaches past the target family's view-angle limit: the family cannot represent the source there, and it is never fitted over a clipped field. Its fields carry the evidence, so a caller can catch it and record the case (a fisheye × rational cell as not applicable, quoting the angle) without reading the message:
family::Type: the target lens family;field_angle::Float64: the region's largest view angle off the source's axis, in RADIANS (the source's own limit where the region reaches past the radii it images);limit::Float64: the family's view-angle limit, in RADIANS (Core'sview_angle_limitof the family: the perspective plane'sπ/2 − 10⁻³for a gnomonic family, which applies its distortion on that plane).
showerror prints the refusal: for a gnomonic family, "this target applies its distortion on the perspective plane (θ < 90°), but the region reaches <θ>°. Use a lift (division) model, or the same radial factor in an equidistant frame"; for any other, its limit and the region's angle.
try
fit_camera(L, camera, region)
catch e
e isa ViewAngleLimitError || rethrow()
rad2deg(e.field_angle), rad2deg(e.limit) # the measured field and the family's limit, degrees
end