Lens Families

Every lens is one of two kinds, built from a handful of parts. The parts — projections and terms — are leaves, structs of numbers and tuples of numbers; everything a lens is asked (its coefficients, their names, its zero, its record, its manifold, every Jacobian, every batch) is derived from their fields by one walk, never tabulated per model. A coefficient is named by its field — a tuple field k as k1, k2, … — and a leaf's record tag is its name in snake case (DoubleSphere → double_sphere, FieldOfView → field_of_view; the Full radial basis is radial_full).

Two kinds of lens

  • Lens(projection, distortion) — forward: the ray is projected onto the projection's frame, and the frame point is distorted there, x_d = distortion(projection(X)). Projection is closed form; back-projection inverts the distortion.
  • LiftLens(radial) — backward: the image point lifts to its ray in closed form, (x_d·N(ρ), D(ρ)); projection is a root on the ray. D crossing zero is the 90° circle and D < 0 the field behind it.

Projections

projectionradiusframe
Perspectiver = tan θGnomonicFrame
Equidistantr = θEquidistantFrame
Stereographicr = 2 tan(θ/2)StereographicFrame
Equisolidr = 2 sin(θ/2)EquisolidFrame
Orthographicr = sin θOrthographicFrame
DoubleSphere(xi, alpha)Usenko et al. (2018)model-defined
Unified(xi)Mei and Rives (2007) (Kalibr omni)model-defined
ExtendedUnified(alpha, beta)Khomutenko et al. (2016) (Kalibr eucm)model-defined

Lens(projection) alone is the pure projection; Pinhole is Lens(Perspective()).

The whole sphere of directions onto the (longitude, latitude) rectangle — a panorama raster, a sky map — is Equirectangular: not a lens (it has no radial profile), a projection of its own with project/backproject. Its cylindrical family shares the longitude and changes the vertical coordinate: Cylindrical (tan φ, the central projection onto the unit cylinder), CylindricalEqualArea (sin φ: equal raster areas are equal solid angles) and Mercator (asinh(tan φ), conformal). Each is a Transformation whose inv is its Backprojection, so a panorama raster's pixel → ray map composes into an image_warp chain; an azimuthal view needs no type of its own — it is Backprojection(camera) of a virtual camera with the view's lens (Lens(Perspective()), Lens(Equidistant()), Lens(Stereographic())).

Terms and how they combine

termkindfunction of the frame point x, r² = ‖x‖²names
Radial(k, d)mapx·N/D, N = 1 + Σ kᵢ r²ⁱ, D = 1 + Σ dⱼ r²ʲk1…, d1…
Tangential(p1, p2)delta(2p₁uv + p₂(r² + 2u²), 2p₂uv + p₁(r² + 2v²))p1, p2
ThinPrism(sx, sy)delta(Σ sxᵢ r²ⁱ, Σ syᵢ r²ⁱ)sx1…, sy1…
Tilt(tx, ty)mapOpenCV's tilted-sensor projective maptx, ty
FieldOfView(omega)mapx·atan(2r tan(ω/2))/(rω)omega

The arity of a term is its tuple lengths — a type parameter — so a four-coefficient Kannala–Brandt and a six-coefficient one are the same family. Radial{Full} uses the powers ρ², ρ³, … instead of ρ², ρ⁴, … (Scaramuzza's lift).

  • Additive(m, Δ₁, …) — every term at the same point, summed: m(x) + (Δ₁(x) + …), the deltas left to right. OpenCV's projectPoints and COLMAP's Distortion() are of this form.
  • CoordinateTransformations' ∘ — in sequence, (D₂ ∘ D₁)(x) = D₂(D₁(x)) (a term is a Transformation): Aria's Fisheye624 (COLMAP RAD_TAN_THIN_PRISM_FISHEYE) applies its tangential and thin-prism terms to the radially distorted point, Additive(Tangential(…), ThinPrism(…)) ∘ Radial(…); OpenCV's tilt follows its additive model, Tilt(τx, τy) ∘ Additive(…). Each stage is a map term or an Additive. A composition's coefficients are in CT's order, the outer stage's first, and its record lists the stages the same way ("compose", the outer first; the retired "chain", which listed them in application order, is refused).

Whether a lens is radially symmetric about the principal point — what RadialCamera(::PosedCamera) needs — is the trait RadialSymmetry, read off the lens's type: the projections, Radial, FieldOfView and LiftLens are, and an Additive, a ∘ composition or a Lens when every piece is. It fails closed: a lens type defined downstream is asymmetric until it declares itself, VisualGeometryToolkitCore.Cameras.RadialSymmetry(::Type{MyLens}) = VisualGeometryToolkitCore.Cameras.RadiallySymmetric().

Familiar models

modellens
pinholePinhole() = Lens(Perspective())
OpenCV 4/5/8/12 coefficientsLens(Perspective(), Additive(Radial((k1, k2, k3), (k4, k5, k6)), Tangential(p1, p2), ThinPrism((s1, s2), (s3, s4)))) — only the terms in use
OpenCV 14 (tilted)Lens(Perspective(), Tilt(τx, τy) ∘ Additive(…))
OpenCV fisheye, COLMAP OPENCV_FISHEYEKannalaBrandt(k1, k2, k3, k4) = Lens(Equidistant(), Radial((k1, k2, k3, k4)))
COLMAP SIMPLE_RADIAL, RADIALLens(Perspective(), Radial((k,))), Radial((k1, k2))
COLMAP THIN_PRISM_FISHEYELens(Equidistant(), Additive(Radial((k1, …, k4)), Tangential(p1, p2), ThinPrism((sx1,), (sy1,))))
COLMAP FOVLens(Perspective(), FieldOfView(ω))
division (Fitzgibbon), COLMAP DIVISIONDivision(k…) = LiftLens(Radial((), (k…)))
BabelCalib rationalLiftLens(Radial((a…), (b…)))
Scaramuzza / MATLAB fisheyeScaramuzza(a2, a3, a4) = LiftLens(Radial{Full}((), (a2, a3, a4)))
double sphere (Kalibr ds)Lens(DoubleSphere(ξ, α))
Kalibr omni-radtanLens(Unified(ξ), Additive(Radial((k1, k2)), Tangential(r1, r2)))

The adapters that read and write these conventions — and reproduce each convention's function exactly — live in VisualGeometryToolkitIO.

Coefficients

parameter_vector(lens) is the coefficient vector — projection first, then the distortion's terms depth-first, left to right; parameter_names names it, parameter_bounds states its domain (unbounded, but the double sphere's ξ > −1 and α ∈ [0, 1], the unified models' ξ > −1, α ∈ [0, 1] and β ≥ 0, the field-of-view ω ∈ [0, π)), parameters pairs the names and values in a NamedTuple, and with_parameters rebuilds the same structure from a vector of any number type. The same functions describe a calibratable target, its TargetCalibration, a LogicalIntrinsics (f, aspect, skew, cx, cy, with f, aspect > 0) and a parametric CameraModel — the intrinsics' coordinates followed by the lens's. ParameterManifold (a lens's LensManifold, minimal_manifold(lens) or minimal_manifold(camera)) moves the coefficients inside their bounds — a log for a one-sided bound, a scaled logit for an interval — and restrict(ch, names) frees a subset by name: a camera's restrict(minimal_manifold(camera), (:f, :cx, :cy, :k1)) fixes its skew and aspect. An optimizer's search box is not a model fact; it stays the solver's option, keyed by parameter_names.

A caller holds parameters with ONE keyword, fixed: the parameters held, by name or by group (:k for k1, k2, …), () by default (everything is refined; a camera's skew, the gauge, is held by construction, so naming it changes nothing). LensSpec(family; fixed) and VisualGeometryToolkitEstimation's refine_camera_and_poses take it, and free_parameter_names resolves it to the names that stay free — free_parameter_names((:f, :aspect, :skew, :cx, :cy, :k1, :k2), (:aspect, :k)) is (:f, :cx, :cy). A name or group that names nothing of the model, or a parameter named twice, is an ArgumentError, and so is the retired free (the complement), naming fixed; the refines refuse fix, fixed_params and hold alike.

radial(lens) is the lens's radially symmetric factor — its folds and poles bound what the lens images (view_angle_limit, monotone_radius) — and radial_part the lens with only that factor.

The valid domain and the image circle

Two regions of the sensor, easy to confuse and never interchangeable:

  • valid_domain(camera) is where the lens MODEL is valid: the image of the cone of view_angle_limit through the radial projection and K, inside which the lens's radial profile is finite and strictly increasing. It is a property of the formula and says nothing about light: a fitted Kannala–Brandt profile can fold far outside the lit disc.
  • image_circle(camera, diameter) is where light falls, from a lens datasheet's image-circle diameter through the sensor's pitch. A circle measured from images serves equally; it is never predicted from the model.

For a radially symmetric lens both are an Ellipse in pixels about the principal point (an ellipse proper for an anisotropic K or pitch), so the region algebra treats them alike.

  • For a lens that is not radially symmetric (a composed distortion with a tilt stage, tangential or thin-prism terms) the fold radius depends on the azimuth, and the valid domain is a PolygonRegion: the star polygon through each azimuth's own fold, in the same region algebra. A pixel-frame map moves it vertex by vertex (VisualGeometryToolkitIO's region body writes it as a polygon), and Adapt.adapt moves it to a device, where p ∈ v runs inside a kernel.
  • A lens with no limit of its own (a perspective lens, an undistorted equidistant lens: view_angle_limit reports only its chart's or the sphere's edge, a cap) has the WholePlane as its valid domain, the identity of ∩.
using VisualGeometryToolkitCore
using VisualGeometryToolkitCore: Pinhole, KannalaBrandt
s = sensor("Sony", "IMX250")                            # 2448 × 2048 @ 3.45 μm
k = -4 / (3π^2)                                         # r = θ(1 + kθ²) folds at 90°, r = (2/3)θ
f = ustrip(px, 6.0mm / (2 * s.pitch.width)) / (π / 3)   # a 180° lens drawing a 6 mm circle
cam = BoundedCamera(CameraModel(CameraCalibrationMat(f, Rect(s)); lens = KannalaBrandt(k)), s)
v = valid_domain(cam)                                   # where the lens formula holds
(centroid(v), v.a, v.b)
([1224.0, 1024.0], 869.5652165217391, 869.5652165217391)
q = image_circle(cam, 6.0mm)                            # where light falls: the datasheet, through the pitch
(v.a - q.a, Rect(extent(cam)) ∈ q, rad2deg(angle_of_view(cam)), rad2deg(angle_of_view(cam; dimension = :diagonal)))
(-8.695652695678291e-7, false, 179.9999998199235, 179.9999998199235)

Here the lens was chosen so the two coincide; for a fitted lens they need not. angle_of_view is the angle of view the camera images — its extent bounded by its lens coverage, clipped by the model's valid domain; field_of_view is the size that angle spans at a distance.

rcam = BoundedCamera(CameraModel(CameraCalibrationMat(2000.0, Rect(s)); lens = Pinhole()), s)
(valid_domain(rcam), rad2deg(angle_of_view(rcam)), rad2deg(angle_of_view(rcam; dimension = :diagonal)))
(VisualGeometryToolkitCore.Primitives.WholePlane(), 62.933264526160734, 77.17464549362784)

A local lens model — a grid of nodes rather than a formula — has observations only where light falls: the camera's imaged_region, the sensor ∩ its lens coverage (here the datasheet's circle) ∩ where the lens model is valid:

domain = imaged_region(with_lens_coverage(cam, q))         # a grid lens's domain
nodes = [Point2(x, y) for x in range(0.0, 2448.0; length = 25), y in range(0.0, 2048.0; length = 21)]
lit = filter(n -> n ∈ domain, nodes)
(length(lit), length(nodes) - length(lit))              # lit, outside
(225, 300)

The whole example, runnable, is examples/image_circle.jl.

Which parameters a camera can see

A camera images a region of its sensor, imaged_region(camera) = Rect(extent) ∩ lens_coverage(camera) ∩ valid_domain(camera), its lens coverage (the image circle it carries, lens_coverage); pixel_mask(region, width, height) rasterizes it, the pixels whose centres lie in it. Each parameter of a model has a support on the sensor; it is free when its support meets the imaged region and frozen otherwise. A global model (a parametric lens, K) has the whole plane as every parameter's support, so free_nodes answers Colon() (all of them); a node model such as a SplineLens camera is LocalSupport and only the nodes whose footprint meets the lens coverage are free.

fish = CameraModel(CameraCalibrationMat(250.0, Size2(1280, 960)); lens = KannalaBrandt(k))   # principal point at the image centre
g = CameraModel(SplineLens, fish, Size2(1280, 960); grid_size = (20, 16))
circle = Ellipse(Point2(640.0, 480.0), 250.0, 250.0, 0.0)   # a measured image circle, 250 px
R = imaged_region(BoundedCamera(fish, Size2(1280, 960); lens_coverage = circle))
mask = node_mask(g, R)                                  # in the grid's shape
lit = pixel_mask(R, 1280, 960)                          # the pixels it images: 960 × 1280, ≈ π · 250² of them
(size(mask), count(mask), length(mask) - count(mask), free_nodes(fish, R), count(lit))   # free, frozen; a lens: all
((16, 20), 96, 224, Colon(), 196364)

A lattice model declares its supports through two contract generics: support_trait(::Type{<:MyLattice}) = LocalSupport(), then VisualGeometryToolkitCore.nodes(m) (the node ids, CartesianIndices of the lattice) and VisualGeometryToolkitCore.support(m, i) (the node's region on the sensor in pixels — for a uniform cubic B-spline lattice cubic_bspline_footprint(center, spacing)); free_nodes and node_mask then work unchanged.

A spline lens in a refinement

A refinement of a SplineLens camera reads it through a few primitives of its lattice. The controls of a lens may be kept off the sphere (unit_controls = false): backproject normalizes the spline value, so a control's norm is an interpolation weight, not a gauge — an optimizer stepping in each control's tangent plane (direction_tangents) leaves them there. The lens's backward is the raw spline, a ChartSpline on the normalized plane x = K⁻¹u (generic in the element type, Float32 on a device), continued past its area by the outermost cells' cubic pieces; a per-tile cost block evaluates a window — the camera of a block of controls, possibly replaced, which evaluates as the camera wherever its windows lie in the block; spline_window gives the 4 × 4 controls and weights a point reads, for the Jacobian's fan-out onto the controls; and spline_value lays the same lattice under any control type (a scalar field over the image).

init = :quasi_interpolate builds a camera from any source — a camera or a function pixel -> direction (nothing where it images nothing) — by the local 4th-order quasi_interpolant on a lattice extended one node past the area, linearly extrapolating per axis where the source images nothing and inward from the area's centre where its domain ends inside the area. A camera source keeps its own K; a function source takes the gauge of its sampled spline (the principal point is the pixel imaging ẑ). Where light falls is the bound's, not the lens's: BoundedCamera(cam, extent; lens_coverage = circle) carries a supplied image circle, and free_nodes(cam, imaged_region(bc)) are the nodes that can reach a pixel it images.

A lattice already known — Schöps et al.'s intrinsics0.yaml, a stored grid record — is built from its controls: CameraModel(SplineLens, controls, size; calib_area, intrinsics, present) lays the direction controls ([gy, gx]) over the calibrated pixel area as the reference does, so every pixel backprojects as its grid does; K is the given one (skew 0) or the lattice's gauge, and an absent node's control is never read. It is the one constructor: the source constructor and spline_lattice build through it.

A lens need not hold the other nodes. Built with with_present(cam, node_mask(cam, region)), it keeps the same lattice (spacing, basis, the dense control matrix) and holds only those nodes; every pixel of the region reads them alone, so the camera evaluates the region as the full one. present_nodes is the mask; past the fully present windows backproject is NaN and project gives a NaN pixel, as outside the area. with_directions(cam, D) replaces the direction controls (the same K, lattice and mask; the forward spline, which inverted the old directions, is dropped). Both rebuild a BoundedCamera too, keeping its bound, lens coverage and pose.

A spline's ẑ is a gauge: wherever its first calibration's frame put it, which may be far from the lens's optical axis. axis_gauged(cam, R, K′) turns the camera frame by R (every control R·c, so every ray R·d, exactly) and takes the skew-0 K′ as its gauge (no ray changes). Turn the poses by the same R with axis_gauged(extrinsics, R); a posed camera turns its own. Every image is then kept. With R taking the optical axis onto ẑ and K′ the radially symmetric camera fitted to the lens (its principal point the axis's pixel), the principal point is the optical axis.

cam = CameraModel(SplineLens, fish, Size2(1280, 960); grid_size = (35, 25), init = :quasi_interpolate)
w = window(cam, 5:14, 15:24)                       # rows (y) × cols (x) of the grid
backproject(w, SVector(700.0, 400.0)) ≈ backproject(cam, SVector(700.0, 400.0))

Extending and subdividing a lattice

Two host-side operations put a SplineLens on another lattice without changing it where it is defined. Resampling one spline lens onto a target lattice composes them: extend to cover the target's area, then subdivide towards its spacing.

ext  = extend_lattice(cam, target_area)                  # (; camera, extrapolated, offset, folds)
fine = subdivide_lattice(ext.camera, (2, 3); ext.extrapolated)   # (; camera, extrapolated)

Subdivision is exact. On a uniform cubic lattice, node j sits at cell j − 2. Its basis is the cardinal cubic B-spline M(t − j + 4), with M supported on [0, 4]. Splitting every cell into m equal cells inserts the knots k + i/m, 0 < i < m, into every span. Inserting them one at a time is Boehm's algorithm, and inserting them all at once has a closed form, the two-scale relation of M:

\[M(t) = \sum_{k=0}^{4(m-1)} a_k\, M(mt - k), \qquad \sum_k a_k z^k = m^{-3}\,(1 + z + \dots + z^{m-1})^4 .\]

It follows from M being the four-fold convolution of the unit box B on [0, 1). The box splits into m boxes, B(t) = Σᵢ₌₀^{m−1} B(mt − i), and a convolution of two functions of mt is 1/m times the function of mt of their convolution. Four boxes give the mask (1 + z + … + z^{m−1})⁴ and three convolutions the factor m⁻³. Coarse node j is therefore Σₖ aₖ times the fine node j′ = m(j − 4) + k + 4. The fine controls are d = R c per axis, with R the (m(n − 3) + 3) × n matrix of those aₖ. On the lattice, D = R_y C R_xᵀ.

  • The coefficients of each residue class mod m sum to one, so a constant is reproduced.
  • The fine nodes 1 … m(n − 3) + 3, those whose support meets the area, read coarse nodes 1 … n only. So the fine spline equals the coarse one on the whole closed area.
  • Each fine edge cell is a piece of the coarse edge cell's cubic. So the continuation past the area, which runs on the edge cell's cubic, is the same too.
  • The representation in the refined space is unique, so this is exactly what Boehm's insertion gives. The tests check it against a literal insertion.

A fine node is present when every coarse node it reads is present. A fine window inside a present coarse cell reads only coarse nodes of that cell's window, so it is present. A coarse cell with an absent node has, in each of its fine cells, a fine node that reads that node, since every aₖ > 0. The imaged region and the valid_domain are therefore the source's.

Extension keeps the source bit for bit. The lattice grows by whole cells on its own spacing, so its origin and phase are the source's. The new node count per side is set by the region: a node at cell t is needed when its open support (t − 2, t + 2) meets the region. Every present source node keeps its control. Every window inside the source's area reads source nodes only, so the spline there is the source's, to the rounding of the lattice coordinate.

The other needed nodes, new or absent in the source, are extrapolated. Their controls minimise the discrete bending energy of the extended lattice, with the present source nodes held:

\[E(c) = \sum \frac{\Delta_{xx}c^2}{h_x^4} + 2\,\frac{\Delta_{xy}c^2}{h_x^2 h_y^2} + \frac{\Delta_{yy}c^2}{h_y^4} ,\]

per component, on cells h_x × h_y pixels. The rows are difference_rows(nodes, 2) along each axis and its mixed rows (axes = (), mixed = √2), each block weighted by its cells' widths, and the minimiser is one sparse Cholesky solve. An absent source node that stays absent is not part of the lattice the energy sums over.

  • Continuity. A uniform cubic B-spline is C² at every knot, whatever its controls, and the source's domain edge is a knot. So the extended spline is C² across the edge: its one-sided value, slope and curvature there are the source's.
  • Rotation invariance.E is the thin-plate energy f_xx² + 2f_xy² + f_yy² in pixels, discretized. It weighs the normal, tangential and mixed curvature alike, on square and non-square cells, so none of them can grow outward, corners included.
  • Null space. The null space of E is the affine functions of the node indices. An affine lattice is continued exactly, and a lattice without tangential variation is continued by the natural rule c₀ = 2c₁ − c₂ (natural_extrapolation), whose curvature vanishes one cell out.
  • Uniqueness. The minimiser exists and is unique when one complete 4 × 4 window of the source is present and every extrapolated node is read by some difference: an affine function that vanishes on a 4 × 4 block vanishes everywhere. Without such a window, extend_lattice refuses.

Folds. The extension is C² and its curvature does not grow, but it is not guaranteed injective. A source whose own rim is already strongly curved continues that curvature into the band, and the band can fold: a Schöps grid past its lens coverage is often non-injective itself. folded_pixels(camera, region; spacing) lists the samples of a region where the backward map's orientation d · (∂d/∂u × ∂d/∂v) turns, and extend_lattice returns it over the requested region as folds. Core reports a fold and never refuses one: the caller sets the policy.

The penalty gives only the START values of the extrapolated nodes. No refinement's objective holds it, and a later fit moves those nodes by its own terms. The extrapolated mask says which they are. A subdivision carries the marks: a fine node is marked when it reads a marked coarse node. A marked coarse node is new, so its support lies outside the source's area, or absent in the source, so the source images none of its cells. The pixels the source images therefore read unmarked nodes only, before and after the subdivision.

The lens record

lens_record writes a lens as a self-describing record, the form VisualGeometryToolkitIO stores and configurations name: every key a leaf tag or a combinator, every value a coefficient array whose length is the arity.

lens_record(KannalaBrandt(-0.043, 0.011))
# {"equidistant": [], "radial": [[-0.043, 0.011], []]}
lens_record(Division(-0.31, 0.024))
# {"lift": {"radial": [[], [-0.31, 0.024]]}}
lens_record(Lens(Perspective(), Additive(Radial((-0.28,)), Tangential(1.0e-3, -5.0e-4))))
# {"perspective": [], "add": [{"radial": [[-0.28], []]}, {"tangential": [0.001, -0.0005]}]}

lens_from_record inverts it exactly: the record determines the lens type, and with shortest round-trip floats a Float64 lens reads back bitwise.