Cameras API

Cameras, camera models, intrinsics, and sensors. The grid cameras (a SplineLens, a NoncentralSplineModel) are on the Spline Lens API page; calibration (the radial camera, intrinsics from homographies, view-angle limits) is on the Calibration API page; lens models and the lens contract with its batched surfaces are on the Lenses API page; the framed imaging chain, ray maps, and stereo rigs are on the Imaging Chain and Stereo API page. See the Cameras guide.

Cameras and Camera Models

VisualGeometryToolkitCore.Cameras — Module
Cameras

Camera system submodule for VisualGeometryToolkitCore.

Provides complete camera modeling including:

  • Sensors: Physical CMOS sensor specifications
  • Intrinsics: Calibration matrices and intrinsic parameters (physical/logical)
  • Lens Models: forward lenses (Lens: a projection + a distortion in its frame) and lifts (LiftLens)
  • Camera Models: Composable camera models (intrinsics + lens)
  • Cameras: Complete cameras in 3D space (model + pose)
  • Stereo: Stereo camera rigs and epipolar geometry
  • Utilities: Helper functions (focal_length, lookat, P3P solvers)

Main Types

  • PosedCamera, PosedPinholeCamera
  • CameraModel
  • LogicalIntrinsics
  • Lens, LiftLens, Pinhole, KannalaBrandt, Division
  • Sensor, CMOS_SENSORS
  • StereoRig

Main Functions

  • project, backproject, unproject
  • focal_length, lookat, pose
  • p3p, sample_p3p
  • default_frustum_depth

Example

using VisualGeometryToolkitCore
using VisualGeometryToolkitCore.Cameras

s = sensor("Sony", "IMX174")
camera = BoundedCamera(8mm, s; lens = Lens(DoubleSphere(0.25, 0.4)))   # a lens on the sensor, centred
f = focal_length(s, deg2rad(40.0))                                     # or the focal length of a field of view
model = CameraModel(f, s.pitch, centroid(Rect(s)) .* px)               # the general form: any principal point
posed = PosedCamera(model, lookat(...))
source
VisualGeometryToolkitCore.Cameras.BoundedCamera — Type
BoundedCamera{C<:AbstractCamera, B, R} <: AbstractCamera

A wrapper around any AbstractCamera that adds pixel-space image bounds (width × height) without disturbing the underlying camera's intrinsics or units. Used by SfM/MVS dataset readers (ETH3D, ScanNet, COLMAP exports) where the camera is described by its calibration plus an explicit image extent — information the bare PosedCamera type doesn't retain.

The wrapped camera may be UNPOSED — a bare model, is_posed(bc) == false — which is COLMAP's "camera" row (intrinsics, lens and image size, no pose) and the element of a stored camera collection (VisualGeometryToolkitIO's cameras kind). BoundedCamera(PosedCamera(model(bc), pose), bc.bound; lens_coverage = lens_coverage(bc)) is its posed counterpart; the pose accessors of an unposed one are an ArgumentError. is_posed is the wrapped camera's.

The bound

BoundedCamera(camera, extent::Size2) bounds by a pixel extent; BoundedCamera(camera, sensor::Sensor) by the physical sensor, whose resolution is the extent and whose pitch (a length per pixel) carries a lens datasheet's millimetres into pixels (image_circle). Either way extent(bc) is the Size2{Int} pixel extent; sensor(bc) is the Sensor, or nothing for the extent form. The bound is ONE field, bc.bound: the extent, or the sensor whose resolution is the extent.

The lens coverage

BoundedCamera(camera, bound; lens_coverage = region) — where light reaches the sensor, in pixels: a Circle or an Ellipse (the lens's image circle, measured with a detector or from a datasheet), or a Rect2; any real element type, converted to Float64, and kept as the kind given. Omitted, it is the sensor's own Rect: the whole sensor is lit. lens_coverage(bc) reads it, with_lens_coverage replaces it. What the camera images is imaged_region(bc): the sensor ∩ the lens coverage ∩ where the lens model is valid.

Field access

  • bc.camera, bc.bound, bc.lens_coverage — direct fields; read them through model, extent/sensor and lens_coverage.

Method delegation

A BoundedCamera is THE type for pixel operations: every camera verb a CameraModel answers is forwarded to the wrapped camera (MacroTools.@forward, and @inlineproject/backproject for device kernels) — the maps and their Jacobians (project, backproject, backproject_with_jacobian, unproject, is_in_front, constraint_with_jacobians), the accessors (intrinsics, lens, pose, extrinsics, focal_length, model, calibration_matrix), the imaging and its domain (imaging_transform, imaging_chain, distortion_map, valid_domain, valid_rect, undistort, …) and the camera-derived solvers (p3p, transfer, radial_homography); so backproject_with_jacobian(bc, u) == backproject_with_jacobian(bc.camera, u). The bound applies to none of them (the extent is image_circle, imaged_region and lens_coverage). The model constructors (pinhole, window, with_parameters) and the parameter protocol are the model's: call them on model(bc), and rebound with with_model.

Trait

bounds_trait(::BoundedCamera) = HasBounds() — methods that need image extent dispatch on BoundsTrait: extent(bc) is the bound's pixel extent.

A device kernel's camera

The bound's kind and the region's are the type parameters B (Size2{Int}, or the stored Sensor type) and R (Rect2{Float64}, Circle{Float64} or Ellipse{Float64}), so a bounded camera is concrete, and isbits whenever the wrapped camera is (a CameraModel of an isbits lens): it travels into a GPU kernel as it is, its backproject/project the wrapped camera's. convert_eltype(Float32, bc) converts the wrapped camera and keeps the bound and the region.

Example

inner = PosedCamera(CameraModel(K; lens=Pinhole()), pose)
bc    = BoundedCamera(inner, Size2(6048, 4032))
intrinsics(bc)     # forwarded → CameraCalibrationMat
extent(bc)         # Size2(6048, 4032)
lens_coverage(bc)     # Rect2(0, 0, 6048, 4032): the whole sensor is lit
source
VisualGeometryToolkitCore.Cameras.BoundedCamera — Method
BoundedCamera(f::Quantity, sensor::Sensor; lens = Pinhole(), lens_coverage = nothing) -> BoundedCamera

A lens of focal length f (a length, 8mm) on sensor: the camera model of focal length f / pitch pixels with its principal point at the sensor's centre (centroid(Rect(sensor))), bounded by the sensor, so its pitch stays with it (metric_focal_length). The camera of a datasheet, its lens model lens. A principal point off the centre is the general form, BoundedCamera(CameraModel(f, sensor.pitch, pp; lens), sensor).

sensor = Sensor(Size2(320, 240), 20μm)
camera = BoundedCamera(8mm, sensor; lens = Lens(DoubleSphere(0.25, 0.4)))   # 400 px, principal point (160, 120)
source
VisualGeometryToolkitCore.Cameras.OrientedCamera — Type
OrientedCamera{M}

A camera that only turns about its centre: a PosedCamera whose extrinsics are a RotationMap{3}, PosedCamera{M, <:RotationMap{3}} (an alias, not a type of its own). It has an orientation and no position: extrinsics(c) is the world → camera rotation, pose(c) its inverse, and project, backproject and unproject act about the origin. Build it from the rotation of either direction: PosedCamera(model, relabel(CameraFrame, WorldFrame, RotationMap(R))) takes a camera → world orientation and inverts it, an unlabeled or world → camera RotationMap is stored as the extrinsics. What needs a position refuses one by name: a rig member, the plane-induced homography and the verbs on plane coordinates (project of a plane conic or rectangle, transfer_ellipse, HomographyMat, RadialCamera), and a non-central grid model's pose. The frames of a panorama are oriented cameras, as view_overlap takes them.

source
VisualGeometryToolkitCore.Cameras.PosedCamera — Type
PosedCamera{M, Rt} <: AbstractCamera

A camera in 3D space, consisting of a camera model (intrinsics + lens) and extrinsics (pose).

Dispatch on the lens through the model parameter: PosedCamera{<:CameraModel{<:Any, <:MyLens}} (the PosedPinholeCamera alias is that form for Pinhole).

Use project(camera, X) to map a 3D world point to 2D pixel coordinates, and backproject(camera, u) / unproject(camera, u, depth) for the inverse directions. The transform is computed on demand and inlined.

Type parameters

  • M <: AbstractCamera: an UNPOSED camera model (CameraModel, or a grid model); PosedCamera(model, X) refuses a posed one (is_posed(model)) and a BoundedCamera — the bound goes outside the pose, BoundedCamera(PosedCamera(model, X), extent)
  • Rt <: Union{EuclideanMap, RotationMap{3}}: extrinsics type. A RotationMap poses a camera that only turns about its centre, an OrientedCamera; a camera turns in three dimensions, so a RotationMap{2} is a MethodError at construction.

Fields

  • model::M: Camera model (intrinsics + lens)
  • extrinsics::Rt: Camera extrinsics (world-to-camera transform)

Examples

# Create camera with logical intrinsics
model = CameraModel(800.0px, [320.0px, 240.0px])
extrinsics = EuclideanMap(RotMatrix(I), [0.0, 0.0, 0.0])
camera = PosedCamera(model, extrinsics)

# Dispatch on lens (via the model parameter; PosedPinholeCamera is the
# ready-made alias for the Pinhole case)
foo(::PosedPinholeCamera) = "pinhole path"
foo(::PosedCamera{<:CameraModel{<:AbstractIntrinsics, <:KannalaBrandt}}) = "fisheye path"

# Project a world point
X_world = SVector(1.0mm, 2.0mm, 10.0mm)
u = project(camera, X_world)
source
VisualGeometryToolkitCore.Cameras.PosedCamera — Method
PosedCamera(model::CameraModel, e::CameraExtrinsics) -> PosedCamera
PosedCamera(model::CameraModel, p::CameraPose) -> PosedCamera

Construct a PosedCamera from a frame-labeled map, so the caller states which frame it holds instead of relying on the positional convention.

The bare positional constructor PosedCamera(model, extrinsics::EuclideanMap) with an UNLABELED map takes the world-to-camera map (documented convention, unchecked); a caller holding a camera-to-world map (a lookat-style pose, a renderer's sensor.to_world) who supplies it positionally stores the WRONG map without any error — every accessor then silently answers for the inverse camera. With frame labels that mistake is unrepresentable: CameraExtrinsics (being the extrinsics-labeled EuclideanMap) stores directly into the extrinsics slot via the default constructor, CameraPose inverts internally, and pose(PosedCamera(model, P::CameraPose)) ≈ P.

source
VisualGeometryToolkitCore.Cameras.PosedCamera — Method
PosedCamera(K::StaticMatrix{3,3}, R::StaticMatrix{3,3}, t::StaticVector{3};
       check=true, project=false) -> PosedCamera

Boundary constructor taking raw matrix data (typical for SfM dataset loaders). R is wrapped via the EuclideanMap(::SMatrix, ...) path which validates (and optionally snaps via project=true) before promoting to RotMatrix. t is plain numbers of K's element type, or lengths (t .* m), which give a physical camera.

source
VisualGeometryToolkitCore.Cameras.PosedPinholeCamera — Type
PosedPinholeCamera

Cameras whose lens is exactly Pinhole — the alias for lens dispatch now that PosedCamera carries only {M, Rt} (the former {I, L, Rt, M} repeated what M already determines). Consumed by the induced-homography constructors in geometry/cameras/homography.jl.

source
VisualGeometryToolkitCore.Cameras.calibration_matrix — Method
calibration_matrix(camera::PosedCamera) -> SMatrix{3,3,Float64,9}

Return the 3×3 calibration matrix K as a plain SMatrix. For a SplineLens camera K is the lens's gauge — the pinhole at the pixel imaging ẑ (skew 0) — kept as it is by every fit.

Example

camera = PosedCamera(CameraModel(; focal=800.0, pp=(320.0, 240.0)), ext)
K = calibration_matrix(camera)  # [800 0 320; 0 800 240; 0 0 1]
source
VisualGeometryToolkitCore.Cameras.is_in_front — Method
is_in_front(camera, x, y, z) -> AbstractArray{Bool}

Broadcasted component-array (SoA) form of the chirality test: world-frame point components in, camera-frame z > 0 mask out. Per-element semantics match is_in_front(camera, Point3(xᵢ, yᵢ, zᵢ)); the component layout keeps the whole test one fused broadcast on GPU arrays (hoisted from the RayTracedCalibration GPU pipeline).

source
VisualGeometryToolkitCore.Cameras.is_in_front — Method
is_in_front(camera, point::Point3) -> Bool

Chirality test: returns true if point is in front of the camera (z > 0 in camera coords).

Broadcasts over points:

mask = is_in_front.(Ref(camera), world_points)
source
VisualGeometryToolkitCore.Cameras.lens_coverage — Method
lens_coverage(camera::BoundedCamera, rim_margin::Real) -> Union{Rect2{Float64}, Circle{Float64}, Ellipse{Float64}, Nothing}

The camera's lens_coverage less rim_margin pixels at the LENS's edge (an image circle's soft rim), or nothing when nothing is left: a Circle of radius r − rim_margin, an Ellipse with both semi-axes less rim_margin, a Rect inset by rim_margin on each side that lies inside the sensor — a side on or past the sensor's edge is the sensor's, not the lens's, and stays, so a camera whose whole sensor is lit loses nothing. The kind it was given; a margin of 0 returns the coverage. Where a consumer counts a pixel as lit away from the rim: imaged_region(with_lens_coverage(camera, c)) is what such a camera images (with with_lens_coverage), and a Circle's c.r the radius it reaches from the circle's centre.

source
VisualGeometryToolkitCore.Cameras.lens_coverage — Method
lens_coverage(camera::BoundedCamera) -> Union{Rect2{Float64}, Circle{Float64}, Ellipse{Float64}}

Where light reaches the camera's sensor, in its pixels: the lens's image circle (a Circle or an Ellipse, measured or from a datasheet), or the sensor's own Rect when the whole sensor is lit — the kind it was given, in Float64. Not where the lens MODEL is valid — that is valid_domain; what the camera images is both, inside the sensor: imaged_region(camera).

source
VisualGeometryToolkitCore.Cameras.metric_focal_length — Method
metric_focal_length(camera::BoundedCamera) -> length

The camera's focal length as a length: its fx pixels times the pixel pitch of the sensor it is bounded by (fx · pitch.width, the lens's focal length when the camera is a lens on that sensor, BoundedCamera(f, sensor)). A camera bounded by a bare pixel extent has no pitch: an ArgumentError.

source
VisualGeometryToolkitCore.Cameras.plane_map — Method
plane_map(camera::PosedCamera) -> CoordinateTransformations.Transformation

The board-plane → pixel map of a posed camera, for every camera model: the z = 0 plane of its world (a planar target's design plane, in its units) lifted to 3-D (q ↦ (q, 0), the target surfaces' plane lift), the camera's rigid extrinsics, then its model's projection, project(camera.model, ·) as a map (the forward twin of Backprojection). plane_map(camera)(q) is project(camera, [q; 0]), and transform_deriv(plane_map(camera), q) its 2 × 2 Jacobian with respect to the plane point, the composition's chain rule (the projection's by ForwardDiff through project, a spline model's by the implicit function theorem).

For a central CameraModel the same map, on the plane points in front of the camera, is the plane-induced homography of its pinhole twin, dehomogenised, then its lens's distortion_map composition (a pinhole: the homography map alone, PerspectiveMap() ∘ HomographyMap(camera)), and plane_map returns that form: it is the composition transfer_ellipse(camera, shape) transfers a plane conic through, stage by stage, so a conic's image is the homography's exact conic action, then the lens's local affinity. A plane point behind the camera (Z ≤ 0) has no image (project is NaN or a ray past 90°), and the homography form does not test it. A SplineLens camera whose lattice does not image ẑ (the off-axis gauge, which distortion_map refuses for an undistortion) takes the same form: the gauge's K cancels between the homography and the lens stage, so the map is the lens's own projection of the plane point.

The camera's extrinsics are plain numbers in the plane's unit: a physical camera's plain form is ustrip(m, camera).

source
VisualGeometryToolkitCore.Cameras.sensor — Method
sensor(camera::BoundedCamera) -> Union{Sensor, Nothing}

The physical Sensor a camera is bounded by — its resolution is the camera's extent, its pitch the length per pixel — or nothing for a camera bounded by a bare pixel extent. The one-argument accessor beside the two-argument database lookup sensor(manufacturer, model): both answer a Sensor.

source
VisualGeometryToolkitCore.Cameras.with_lens_coverage — Method
with_lens_coverage(camera::BoundedCamera, region) -> BoundedCamera

The same camera and bound with the lens coverage region (a Rect2, Circle or Ellipse of any real element type, converted to Float64; nothing for the whole sensor).

source
VisualGeometryToolkitCore.backproject — Method
backproject(camera::PosedCamera, u::StaticVector{2, Float64}) -> Vec3

Backproject a 2D pixel to a 3D ray direction in world coordinates (normalized). Returned as a Vec3 because it represents a direction — not translatable under affine maps. The ray's origin is the camera center (pose(camera).t); points along it are pose(camera).t + s · dir for s > 0.

source
VisualGeometryToolkitCore.extent — Method
extent(cam) -> Size2{Int}

Pixel-space image extent (width × height). Trait-dispatched: errors at compile time for cameras that don't carry bounds.

source
VisualGeometryToolkitCore.extrinsics — Method
extrinsics(camera) -> CameraExtrinsics

Camera extrinsics (world-to-camera transform). Returns a frame-labeled map (WorldFrame → CameraFrame): the extrinsics slot is world-to-camera by contract, so the accessor labels it even when the camera was constructed from an unlabeled map (identity when already labeled).

camera = PosedCamera(model, ext)
E = extrinsics(camera)  # World-to-camera
E(world_point)          # Transform point to camera coordinates
source
VisualGeometryToolkitCore.focal_length — Method
focal_length(camera::PosedCamera)

Get focal length from camera's intrinsics: the (fx, fy) tuple in pixels.

Examples

K = CameraCalibrationMat(800.0px, [320.0px, 240.0px])
model = CameraModel(LogicalIntrinsics(K), Pinhole())
camera = PosedCamera(model, extrinsics)
fx, fy = focal_length(camera)  # Returns (800.0, 800.0)
source
VisualGeometryToolkitCore.model — Method
model(camera::PosedCamera) -> CameraModel

Return the camera's underlying CameraModel (intrinsics + lens), the single object the lower-level project/backproject/transfer kernels take. Method form so wrappers (e.g. BoundedCamera) can delegate via @forward. A rig's other member kind answers the same accessor: model(::PosedTarget) is its target.

source
VisualGeometryToolkitCore.pose — Method
pose(camera) -> CameraPose

Camera pose (camera-to-world transform), inverse of extrinsics. Returns a frame-labeled map (CameraFrame → WorldFrame).

camera = PosedCamera(model, extrinsics)
P = pose(camera)       # Camera-to-world
P.t                    # Camera position in world coordinates
source
VisualGeometryToolkitCore.project — Function
project(camera::PosedCamera, rect::HyperRectangle{2}, n::Int=20) -> Polygon

Project a 2D rectangle on the world z=0 plane, tessellating each edge into n samples.

source
VisualGeometryToolkitCore.project — Method
project(camera::PosedCamera, X_world::AbstractVector) -> Point2

Project 3D point from world coordinates to pixel coordinates.

Arguments

  • camera::PosedCamera: Camera with model and extrinsics
  • X_world::AbstractVector: 3D point in world coordinates (with units)

Returns

  • Point2 with pixel coordinates (px units)

Example

model = CameraModel(16.0mm, Size2(width=5.86μm/px, height=5.86μm/px), [320.0px, 240.0px])
extrinsics = lookat(SVector(0.0, 0.0, 10.0), SVector(0.0, 0.0, 0.0), SVector(0.0, 1.0, 0.0))
camera = PosedCamera(model, extrinsics)
X_world = SVector(1.0mm, 2.0mm, 3.0mm)
u = project(camera, X_world)
source
VisualGeometryToolkitCore.project — Method
project(camera::PosedCamera, c::Circle3D) -> Ellipse or Polygon

Project an arbitrarily-placed 3D circle through the camera. For pinhole cameras, the plane homography induced by the circle's local (u, v) frame gives the analytic image conic; other cases fall back to boundary sampling and return a Polygon.

source
VisualGeometryToolkitCore.project — Method
project(camera::PosedCamera, origin::Point3, dir::Vec3) -> (Point2, Vec2)

Project a 3D origin and displacement through the camera and return the 2D arrow endpoints — the origin's image and the vector from the origin's image to (origin + dir)'s image. Use to plot a 3D normal or direction as an image-space arrow:

origin_2d, vec_2d = project(cam, c.center, c.radius * normal(c))
arrows2d!(ax, origin_2d, vec_2d; color=:magenta)
source
VisualGeometryToolkitCore.project — Method
project(camera::PosedCamera, shape::Union{Circle, Ellipse}) -> Ellipse or Polygon

Project a 2D circle or ellipse that lies on the world z=0 plane through the camera. For pinhole cameras, the analytic conic projection yields an Ellipse; partially-behind-camera cases or distorting lens models fall back to boundary-point sampling and return a Polygon.

source
VisualGeometryToolkitCore.unproject — Method
unproject(camera::PosedCamera, u::AbstractVector{Float64}, depth::Float64) -> Point3

Unproject a 2D pixel at a given camera-space depth (Z along the camera's forward axis) to a 3D point in world coordinates. Returned as a Point3 because it represents a position — translatable under affine maps. Works for both PhysicalIntrinsics (metric units; depth in the chosen metric unit, e.g. mm) and LogicalIntrinsics (unitless pixels; depth carries whatever unit the caller picks for their scene).

source
VisualGeometryToolkitCore.Cameras.DEFAULT_MAX_FOV — Constant
DEFAULT_MAX_FOV

Recommended half-angle cap for flattening a view onto a perspective plane, as the max_fov of undistorted_rect. 70° gives tan θ ≈ 2.75, so the rectified plane stays a small multiple of the frame; past ~80° (tan 5.7) it grows faster than the detail it reveals, and the last degree before 90° is unbounded.

A RECOMMENDATION, not the default: undistorted_rect defaults max_fov to Inf so that it and valid_rect stay bounded by the same rule. The callers that want a viewing cap — plot_rectification — supply this.

source
VisualGeometryToolkitCore.Cameras.CameraModel — Type
CameraModel{I, L} <: AbstractCamera

Camera model combining intrinsics and lens model: an unposed camera (is_posed is false), the parametric intrinsics × lens family. The central grid camera is one of them, a SplineLens; the non-central grid, NoncentralSplineModel, is the other unposed camera.

Use project(model, X) to map a 3D camera-space point to 2D pixel coordinates, and backproject(model, u) / unproject(model, u, depth) for the inverse directions.

Fields

  • intrinsics::I: Camera intrinsics (LogicalIntrinsics)
  • lens::L: Lens model (Pinhole, a Lens, a LiftLens, …)

Examples

# Logical camera with pinhole lens
K = CameraCalibrationMat(800.0px, [320.0px, 240.0px])
logical = LogicalIntrinsics(K)
model = CameraModel(logical, Pinhole())

# Physical camera with pinhole lens
f = 16.0mm
pitch = Size2(width=5.86μm/px, height=5.86μm/px)
pp = [320.0px, 240.0px]
physical = LogicalIntrinsics(f, pitch, pp)
model = CameraModel(physical, Pinhole())

# With Brown-Conrady distortion
model = CameraModel(logical, Lens(Perspective(), Radial((-0.1, 0.01))))

# Project a point
X = SVector(10.0mm, 5.0mm, 100.0mm)
u = project(model, X)
source
VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(map::AffineMap; lens=Pinhole()) -> CameraModel

Construct a logical camera model from a CoordinateTransformations.AffineMap.

The 2×2 linear part encodes focal lengths (and optional skew), the 2D translation encodes the principal point. This constructor enables idiomatic composition:

K = CameraCalibrationMat(M)
model = CameraModel(EuclideanMap(PixelFrame{:opencv}, PixelFrame{:colmap}) ∘ affine_part(K); lens=lens)
source
VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(K::CameraCalibrationMat; lens=Pinhole(), pixel_frame=:colmap) -> CameraModel

Convenience constructor from a calibration matrix K directly.

Wraps K in LogicalIntrinsics internally. The pixel_frame keyword specifies which pixel coordinate convention K is expressed in (:opencv, :colmap, or :julia). The principal point is shifted to COLMAP convention (pixel centers at 0.5) before storage.

Example

K = CameraCalibrationMat(800.0px, [320.0px, 240.0px])
model = CameraModel(K; lens=Lens(Perspective(), Radial((-0.1,))))

# From OpenCV calibration (pixel centers at 0,0)
K_cv = CameraCalibrationMat(M)
model = CameraModel(K_cv; lens=lens, pixel_frame=:opencv)
source
VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(model::CameraModel, K′::CameraCalibrationMat) -> CameraModel

The same camera with intrinsics K′: the lens re-expressed under the new intrinsics matrix, so the pixel map is unchanged. A parametric lens shares the principal point: normalized′ = normalized / s with s = f′/f, so a scale-covariant radial lens transforms its coefficients by the law aᵢ → aᵢ·s^(2i), bⱼ → bⱼ·s^(2j) (the generalized k′ = k·s²); it throws for a moved principal point (the lens center is data, not a frame choice), for anisotropic rescaling of a radial lens (radial symmetry would be broken), and for families whose coefficients are only meaningful under a metric K. A SplineLens takes any skew-0 K′, a moved principal point included (its K is a gauge): its backward area moves by the change of the normalized plane, x′ = x / s + (c − c′)/f′, and so does a forward's output (its chart is about ẑ, which no K moves), so every projection is kept too.

source
VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(f::Quantity, pp::AbstractVector{<:Quantity}; lens=Pinhole()) -> CameraModel

Convenience constructor to build a logical camera model from focal length and principal point (both in pixels).

Example

camera = CameraModel(800.0px, [320.0px, 240.0px])
source
VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(f::Quantity, pitch::Size2{<:Quantity}, pp::AbstractVector{<:Quantity}; lens=Pinhole()) -> CameraModel

Convenience constructor to build a complete physical camera model from parameters.

Example

camera = CameraModel(16.0mm, Size2(width=5.86μm/px, height=5.86μm/px), [320.0px, 240.0px])
source
VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(K::SMatrix{3,3,<:Real}; lens=Pinhole(), pixel_frame=:colmap) -> CameraModel

A camera model from a plain static 3×3 calibration matrix, as dataset readers and pool records store it: K is validated and wrapped as a CameraCalibrationMat, then built by the method above. With the default Pinhole() lens this is the pinhole camera of K. (A plain Matrix goes through CameraCalibrationMat(K) first.)

source
VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(; focal, pp, lens=Pinhole()) -> CameraModel

Unitless convenience constructor for a logical camera model.

Accepts plain Real values (no px units required). Useful for synthetic data generation and quick prototyping.

Arguments

  • focal: Focal length in pixels — scalar Real for isotropic, or Tuple{Real,Real} for anisotropic
  • pp::Tuple{Real,Real}: Principal point (cx, cy) in pixels
  • lens::AbstractLensModel=Pinhole(): Lens model

Examples

model = CameraModel(; focal=800.0, pp=(320.0, 240.0))
model = CameraModel(; focal=(800.0, 805.0), pp=(320.0, 240.0))  # anisotropic
source
VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(m::RadialHomography{T, N}) -> CameraModel{ProjectiveIntrinsics{T}, Division{N, T}}

The projective division camera of a radial homography: CameraModel(ProjectiveIntrinsics(center(m)), Division(division_coeffs(m)...)), built from m's stored primitives. Its ray frame is centred pixels, so the lens coefficients ARE the model's pixel-frame λ's; it drives the fused KA undistort! warp directly: undistort(CameraModel(m), img). No metric ray: a field-of-view operation (a finite max_fov of undistorted_rect) refuses it.

source
VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(f::Tuple{T,T}, pp::AbstractVector{<:Quantity}; lens=Pinhole()) where T<:Quantity -> CameraModel

Convenience constructor to build a logical camera model with anisotropic focal lengths (fx ≠ fy).

Example

camera = CameraModel((800.0px, 805.0px), [320.0px, 240.0px])
source
VisualGeometryToolkitCore.Cameras.backproject_with_jacobian — Method
backproject_with_jacobian(model::CameraModel, u::StaticVector{2}) -> (f, M)

The unit bearing of pixel u and its 3×2 Jacobian M = ∂f/∂u, in ONE evaluation. f is what backproject returns; M is what propagates a PIXEL covariance onto the sphere, so the pair is the entry point for calibrated two-view geometry on bearings (the tangent Sampson residual and its rows).

M is TANGENT by construction: fᵀM = 0 to rounding — a forward lens's is built in a tangent basis, and a backward lens's is the derivative of its unit bearing. Consumers therefore need no explicit projection onto the tangent plane — which is what keeps an inner loop free of the allocation such a projection would cost.

The tangency assertion is LOAD-BEARING — do not simplify it away

fᵀM = 0 is not decorative, and it is not merely an internal consistency check. It is the ONLY test in the suite that catches a correspondence whose two views' Jacobians have been SWAPPED — M₂ handed where M₁ belongs, the natural mistake when porting a residual onto bearings.

Nothing else sees it, and that is measured rather than assumed (test/essential_tangent_mc_tests.jl). The swap is very nearly norm-preserving: the two halves of the tangent Sampson's G_x are scaled by |Eᵀf₂| and |Ef₁|, whose ratio has median 1.018 and 5–95% range [0.972, 1.066], so ‖G_x‖ barely moves and the residual stays plausible. Every Monte Carlo calibration cell keeps reporting NEES ≈ 1 — the best of them gets to 0.970, inside its own 4 SE band. Tangency, by contrast, fails loudly: fᵀM measures 4.8e-19 with the right Jacobian against 1.0e-3 with the swapped one, a factor of 2e15.

So a future cleanup that reads the tangency assertion as redundant with the statistical gates would delete the one check that catches a view swap, and the statistical gates would go on succeeding while it did.

Construction, given that a camera's forward map is the differentiable one: with B an orthonormal basis of T_f S² (s2_tangent_basis, shared with EssentialManifold5 rather than re-derived) and A = ∂project(model, ·)/∂f · B the 2×2 restriction of the projection differential to that plane, M = B·A⁻¹. The projection differential is taken by automatic differentiation through the camera's OWN project, so every forward lens — Kannala-Brandt, double sphere, the OpenCV models — is covered without a per-model derivation, the same derivation-risk-free stance retract_jacobian takes. A BACKWARD lens (a LiftLens — division, Scaramuzza — or a SplineLens), whose back-projection is the closed form and whose projection a solve, instead differentiates backproject itself: M = ∂backproject(lens, x)/∂x · ∂x/∂u at x = K⁻¹u, by the same automatic differentiation, or analytically where the lens has its own (SplineLens).

This is PoseLib's unproject_with_jac; GTSAM's nearest relative is PinholeBase::backprojectPointAtInfinity(p, Dresult_dpose, Dresult_dpoint), which returns a Unit3. GTSAM spells the pixel → calibrated step calibrate, which collides with THIS stack's task verb calibrate (estimating a camera), so the name here stays backproject.

Returns (NaN bearing, NaN Jacobian) wherever backproject does — outside the lens's domain, past a fold, or where the tangent restriction A is singular (a ray the camera cannot resolve in two directions). Never throws.

source
VisualGeometryToolkitCore.Cameras.bearing_correspondences — Method
bearing_correspondences(camera1, camera2, cs_px) -> Correspondences

Pixel correspondences cs_px (u => v, view 1's pixel to view 2's) as bearing correspondences: each endpoint back-projected by its camera (backproject_with_jacobian(camera, u) -> (f, M)), the unit bearings the pair and the 3×2 pixel Jacobians its M₁/M₂ attribute columns, which put a bearing residual back in pixels (bearings.md). One back-projection per endpoint, no filtering: a pixel a camera does not back-project gives its non-finite bearing.

source
VisualGeometryToolkitCore.Cameras.calibration_matrix — Method
calibration_matrix(model::CameraModel{<:LogicalIntrinsics}) -> SMatrix{3,3,T,9}

Return the 3×3 calibration matrix K as a plain SMatrix, preserving the intrinsics' stored element type (Float64 for the default constructors, Float32 for e.g. LogicalIntrinsics(CameraCalibrationMat{Float32}(…))). A projective camera (ProjectiveIntrinsics) has no calibration matrix: there is no method, so the call raises a MethodError. For a SplineLens camera K is the lens's gauge — the pinhole at the pixel imaging ẑ (skew 0) — kept as it is by every fit.

Example

model = CameraModel(; focal=800.0, pp=(320.0, 240.0))
K = calibration_matrix(model)  # [800 0 320; 0 800 240; 0 0 1]
source
VisualGeometryToolkitCore.Cameras.distortion_map — Method
distortion_map(model::CameraModel{<:AbstractIntrinsics, <:AbstractLensModel}) -> Transformation

Math-space transform mapping undistorted COLMAP pixel coordinates to distorted ones: pixel → K⁻¹ → lens → K → pixel. Spelled as a plain composition (affine_part(K) ∘ lens ∘ affine_part(K⁻¹)) so image_warp's fuse merges the calibration affines into the COLMAP/Julia index bridges — the fused per-pixel path is one affine, the lens polynomial, one affine. For a SplineLens camera K is the gauge pinhole at the optical-axis pixel, so the undistorted frame is that pinhole's.

source
VisualGeometryToolkitCore.Cameras.distortion_map — Method
distortion_map(m::RadialHomography) -> Transformation

The radial model's undistorted → distorted pixel map, c + μ·(u − c) with the division model's μ (project(m, (u − c, 1)) as a map): the camera bridge's distortion_map, distortion_map(CameraModel(m)). Its inverse is the undistortion, inv(distortion_map(m)); NaN where the model's field ends (as its lens does), never nothing.

source
VisualGeometryToolkitCore.Cameras.division_bounded — Method
division_bounded(model::CameraModel{<:Union{LogicalIntrinsics, ProjectiveIntrinsics}, <:LiftLens}, sz; margin = HORIZON_MARGIN) -> Bool

Does the whole image lie inside margin · monotone_radius(lens) (in the lens's normalized distorted radius; a projective camera's is centred pixels)? The general-family form of the coefficient-level predicate above.

source
VisualGeometryToolkitCore.Cameras.division_bounded — Method
division_bounded(k, center, sz; margin = HORIZON_MARGIN) -> Bool

Whether the perspective (undistorted) image of a frame of size sz = (width, height) is BOUNDED under a division model k about center (all in the same pixel units — e.g. a RadialHomography's k/center). For k < 0 the undistorted radius r_u = r_d/(1 + k r_d²) diverges at the 90° circle r_∞ = 1/√(−k): content at or beyond it undistorts to infinity, and no finite canvas holds it. True iff the farthest image corner's distorted radius stays inside margin·r_∞ (k ≥ 0 is always bounded).

When true, the corner stretch factor is 1/(1 + k·r_max²) — the canvas scale an undistorted image needs to keep every input pixel in bounds.

source
VisualGeometryToolkitCore.Cameras.fov_radius — Method
fov_radius(model, θ) -> Float64

The UNDISTORTED (perspective-plane) pixel radius of the cone of half-angle θ radians: f·tan(θ). The companion of horizon_radius — one says where the lens stops imaging, this says how big a chosen field of view is once flattened.

A projective camera (ProjectiveIntrinsics) has no focal length, so no field of view: its radius is an ArgumentError (and so is a finite max_fov in undistorted_rect, which reads it).

source
VisualGeometryToolkitCore.Cameras.horizon_radius — Method
horizon_radius(model) -> Float64

The PIXEL radius of the camera's horizon: the circle at which rays reach the lens's monotone limit, beyond which nothing can be placed on a perspective plane at all. Inf for a lens with no such limit (a pinhole, a mild polynomial). This is the circle plot_rectification draws on the raw panel, and the guide for how much of a frame it is meaningful to rectify.

A projective camera (ProjectiveIntrinsics) has no focal length: its radius is read in its own centred-pixel frame, the unit-focal one its lens coefficients are written in.

It is NOT a safe rectification bound by itself. The distorted radius is very nearly linear in the view angle, while the perspective magnification is tan θ, so a generous FRACTION of this radius is still a fraction of a degree from the pole: for a 927 px fisheye with a 1419 px horizon, 92% of it is θ ≈ 89.7°, tan θ ≈ 199, and the rectified plane comes out 369 000 px wide. Bound the ANGLE instead — see undistorted_rect's max_fov.

source
VisualGeometryToolkitCore.Cameras.normalized_point — Method
normalized_point(intrinsics::AbstractIntrinsics, 𝐩::StaticVector{2}) -> SVector{2}

The normalized (distorted) image point x = K⁻¹𝐩 of the pixel 𝐩: the first two coordinates of K⁻¹ (𝐩, 1), K the intrinsics' pixel matrix (skew included). It is the pixel-frame half of backproject — a lens's backproject(lens, x) takes it to the bearing — and the point a spline model's lattice is laid over (spline_window(area, grid, normalized_point(K, 𝐩))). The one spelling of that map: the scalar back-projection, the batched kernels and a caller reading a spline lattice at a pixel all call it. Allocation-free, in the element type of K⁻¹ and 𝐩 (Float32, ForwardDiff Duals), callable in a device kernel.

source
VisualGeometryToolkitCore.Cameras.pinhole — Method
pinhole(model::CameraModel) -> CameraModel{I, Pinhole}
imaging_transform(model::CameraModel)

The lens-stripped pinhole twin (downstream packages hand-rolled CameraModel(x.intrinsics, Pinhole()) in three places) and the composed camera-space → pixel map as a value. intrinsics/lens accessors live in cameras.jl. For a SplineLens camera the twin is the gauge pinhole at the optical-axis pixel (the pixel imaging ẑ).

source
VisualGeometryToolkitCore.Cameras.radial_homography — Method
radial_homography(camera::CameraModel{<:LogicalIntrinsics, <:Division}, H::HomographyMat) -> RadialHomography

The radial homography of a plane homography H (plane point → undistorted pixel) seen through a METRIC division camera: RadialHomography(H, λ, c) with λₚ = kₚ / f^{2p}, kₚ the camera's division coefficients in K's normalized chart, f its focal length (pixels) and c its principal point. The projective method's λ's are pixel-frame; this converts the metric camera's to them, so radial_transfer(radial_homography(camera, H), X) is the raw pixel project(camera, K⁻¹·H·[X; 1]) of a plane point X. Throws an ArgumentError for fx ≠ fy or a skewed K, whose lens is not radially symmetric in pixels.

source
VisualGeometryToolkitCore.Cameras.radial_homography — Method
radial_homography(camera::CameraModel{<:ProjectiveIntrinsics, <:Division}, H::HomographyMat) -> RadialHomography

The radial homography of a target's plane homography H seen through a projective division camera: RadialHomography(H, camera). A TargetHomographies entry (a HomographyMap) pairs with the camera that measured it through the HomographyMap method below.

source
VisualGeometryToolkitCore.Cameras.radial_homography — Method
radial_homography(camera::CameraModel{<:Union{ProjectiveIntrinsics, LogicalIntrinsics}, <:Division}, h::HomographyMap{3}) -> RadialHomography

The radial homography of a plane → pixel HomographyMap (a TargetHomographies entry, labeled or not): radial_homography(camera, H) of its matrix. The map's frames are not checked against the camera, which holds none.

source
VisualGeometryToolkitCore.Cameras.reframed_calibration — Method
reframed_calibration(S::SimilarityMap{2}, K::CameraCalibrationMat) -> CameraCalibrationMat{Float64}
reframed_calibration(from::Type{<:PixelFrame}, to::Type{<:PixelFrame}, K::CameraCalibrationMat) -> CameraCalibrationMat{Float64}

THE change of image frame of a calibration matrix: K in S's source frame becomes to_homogeneous(S)·K in its destination frame, so a pixel K·x̃ of the source is S(K·x̃) of the destination. In Float64. A similarity of the image plane keeps K's form when its rotation is the identity (a scale s multiplies the focals, the skew and the principal point, then the translation moves the principal point); a rotated S gives no upper-triangular K and is refused by K's own structure check (ArgumentError). inv(S) brings K back.

The 3-argument form is the pixel-frame change, S = SimilarityMap(EuclideanMap(from, to)) (a translation by the frames' half-pixel offset; the focal lengths and the skew kept): CameraModel(K; pixel_frame) and VisualGeometryToolkitIO's stored K (its readers and writers) all move a K through it. VisualGeometryToolkitFeatures' half-resolution RAW images move K through the map form with their frame map (raw_frame_map). K was checked (or deliberately not, check = false) when it was built; a positive scale keeps its focals positive, so only the structure is re-checked.

K = CameraCalibrationMat(800.0px, [320.0px, 240.0px])            # OpenCV's frame
reframed_calibration(PixelFrame{:opencv}, PixelFrame{:colmap}, K) # cx, cy = 320.5, 240.5
source
VisualGeometryToolkitCore.Cameras.reprojection_error — Method
reprojection_error(camera, pose, X, u) -> SVector{2}

The reprojection residual of a scene point, Jacobian-free: project(camera, pose(X)) − u, with pose any map into the camera frame (a world → camera EuclideanMap, X_c = R·X + t; a camera → world pose enters as inv(pose), which is x ↦ Rᵀ(x − t)). NaN where the lens cannot image the point (project answers on the ray domain). Element type follows the inputs, so ForwardDiff duals flow through the pose and the lens.

source
VisualGeometryToolkitCore.Cameras.undistort! — Method
undistort!(dst, [T = Float64], model, src, rect; fillvalue = zero(eltype(dst))) -> dst

GENERAL form: resample the undistorted-plane window rect (COLMAP pixel coordinates) into dst. dst's size sets the sampling density — supply a smaller dst to downscale; the scale and offset fold into the fused warp affines. The rect-free methods above render Rect(0, 0, w, h) at unit scale (the classic same-size undistortion, which CROPS any content the undistortion pushes outside the frame); pair this form with undistorted_rect to render the full extent instead.

source
VisualGeometryToolkitCore.Cameras.undistort! — Method
undistort!(dst, [T = Float64], model, src; fillvalue = zero(eltype(dst))) -> dst

In-place undistort: resample src into the pre-allocated dst through the fused lens warp, on the backend that owns dst (any KernelAbstractions backend — supply CuArrays for GPU execution). dst and src may differ in eltype (see image_warp!). T sets the warp math's float type: undistort!(dst, Float32, model, src) keeps the per-pixel math single-precision for GPU throughput. The order is Julia Base's: the mutated dst, then the type, then the inputs.

source
VisualGeometryToolkitCore.Cameras.undistort — Method
undistort(model, img, rect; scale = 1, fillvalue = zero(eltype(img)), device = default_device()) -> FramedImage

Allocating rect form of undistort!: the canvas is sized ceil.(widths(rect) ./ scale), rect is resampled into it, and the result is returned WITH its placement as a FramedImage — rect is the window of the camera's pixel frame the canvas covers, so imshow(pos, undistort(model, img, ext; scale)) is self-placing and the ext/scale pair never travels beside the pixels by hand. The in-place undistort! forms are unchanged (the caller owns dst and its placement); the rect-free same-size forms return bare matrices (their placement is trivially the frame). device as in undistort: the warp runs there, the canvas comes back in img's array kind.

source
VisualGeometryToolkitCore.Cameras.undistort — Method
undistort(model::CameraModel{<:AbstractIntrinsics, <:AbstractLensModel}, img)

Remove lens distortion from an image.

For each output (undistorted) pixel, computes the corresponding source (distorted) pixel through model.lens and resamples. Uses VGC's image_warp builder to bake pixel coordinate conventions into a BoundedMap (K is assumed to be in COLMAP convention: image corner at origin, pixel centers at 0.5).

Each output pixel is sampled through project, which NaNs a ray past a radial factor's fold (the pixel takes the fill value) but does not test the other folds: past a fold of an Additive's deltas or a composed distortion's Tilt the pixel samples a finite source pixel whose backproject is a different ray, so the output repeats content of the imaged branch there. Where the lens overlaps itself over the sensor, no undistortion is one-to-one, and a map built from backproject may jump between preimages (see backproject): injectivity is checked on the lens over the sensor rectangle, not per pixel. A SplineLens camera undistorts onto its gauge pinhole (its K: the pinhole at the pixel imaging ẑ), in every form here and in undistort!.

Arguments

  • model::CameraModel: Camera model (any lens family)
  • img: Input distorted image (any AbstractMatrix — Gray, RGB, etc.)
  • device: the KernelAbstractions backend the warp runs on, by default default_device() (an accelerator when one is functional, else KA.CPU()). img is uploaded to it once, and the result comes back in img's array kind: an Array gives an Array, a device array stays on its device with no copy. The in-place undistort! runs on dst's backend.

Returns

Undistorted image of the same size and element type.

Example

model = CameraModel(; focal=2458.0, pp=(2099.0, 1494.0),
                      lens=Lens(Perspective(), Additive(Radial((-0.159, 0.048)), Tangential(0.006, -0.001))))
img_clean = undistort(model, img)
source
VisualGeometryToolkitCore.Cameras.undistorted_rect — Method
undistorted_rect(m::RadialHomography, (width, height); margin = HORIZON_MARGIN, max_fov = Inf) -> Rect

The undistorted-plane extent of a width×height frame under m's division model (the walk of its CameraModel(m)): the frame boundary undistorted (u = c + (d − c)/(1 + Σₚ λₚ‖d − c‖²ᵖ)), with boundary radii at or beyond margin·r_∞ (the lens's horizon; r_∞ = 1/√(−k) for one coefficient) clipped to it — the honest cap when division_bounded is false and the true extent is infinite. Feed the rect to the rect form of undistort/undistort!:

ext = undistorted_rect(m, (w, h))
canvas = undistort(CameraModel(m), img, ext;
                   scale = max(1, GeometryBasics.widths(ext)[1] / 5000))

max_fov (radians) additionally bounds the extent by FIELD OF VIEW, capping it at the bounding square of that cone about the principal point. Inf, the default, leaves the frame's own field alone; a finite value is what makes a fisheye — whose horizon can sit inside the frame — rectify to a usable plane at all, since tan θ is unbounded in the last degree before the pole. DEFAULT_MAX_FOV is the recommended cap. A field of view needs a metric camera: for a radial homography (a projective camera, ProjectiveIntrinsics) a finite max_fov is an ArgumentError.

source
VisualGeometryToolkitCore.Cameras.valid_rect — Method
valid_rect(m::RadialHomography, (width, height); margin = HORIZON_MARGIN) -> Rect
valid_rect(model::CameraModel{<:Union{LogicalIntrinsics, ProjectiveIntrinsics}, <:AbstractLensModel}, sz;
           margin = HORIZON_MARGIN) -> Rect

The INSCRIBED counterpart of undistorted_rect — OpenCV's alpha = 0 endpoint: the largest axis-aligned window of the undistorted plane containing NO invalid pixels, so undistort(model, img, valid_rect(model, size)) renders with no fill — the no-black-borders mode. Same frame-boundary walk as undistorted_rect (one authority, _boundary_undistort), but taking per-edge extrema INWARD — the left constraint is the maximum x over the undistorted left edge, and so on — instead of the global bounding box (sampled, like the outer rect; corners are the edge-sample endpoints, so both bow directions are covered).

Boundary radii at or beyond margin·r_∞ clip to the horizon exactly as in undistorted_rect: when the whole horizon circle lies inside the frame the valid region is unbounded and the clip is the honest cap. Throws when the inscribed window is empty (boundary folded past the principal point). A SplineLens camera's rect is on its gauge pinhole (the pixel imaging ẑ).

source
VisualGeometryToolkitCore.backproject — Method
backproject(intrinsics::AbstractIntrinsics, lens::AbstractLensModel, u::StaticVector{2,T<:Real}) -> Vec3

The pixel-frame half of back-projection: K⁻¹ takes the pixel u to the normalized distorted point, and the bearing is the lens's own backproject(lens, xd) — which, not this method, is what a lens family extends.

source
VisualGeometryToolkitCore.backproject — Method
backproject(model::CameraModel, pixels::AbstractVector{<:StaticVector{2}}; device = get_backend(pixels)) -> rays
backproject_with_jacobian(model::CameraModel, pixels::AbstractVector{<:StaticVector{2}}; device = get_backend(pixels))
    -> Vector{Tuple{SVector{3}, SMatrix{3, 2}}}

The batched backproject (and backproject_with_jacobian) on the KernelAbstractions backend device (by default the pixels' own, get_backend(pixels): a host vector stays on the host, a device array on its device; give default_device() to send host pixels to the accelerator): the scalar map per pixel, one pointwise! over a device copy of the model and the pixels, each ray an SVector{3}, the result back in the pixels' array kind — bit for bit the scalar value on the CPU. One kernel on every backend: the model goes in its device form (isbits, or its adapt rule); a model with neither fails on an accelerator. A SplineLens camera's batched backproject is its own kernel (spline_lens.jl).

source
VisualGeometryToolkitCore.backproject — Method
backproject(model::CameraModel, u::StaticVector{2,T<:Real}) -> Vec3

Backproject 2D pixel coordinates to 3D ray direction.

Inputs and outputs are unitless (assumed units: px for input, unitless direction for output). The output eltype follows the usual promotion of the input and the intrinsics: Float64 pipelines are unchanged; a Float32 camera (LogicalIntrinsics{Float32}) with Float32 pixels stays Float32; ForwardDiff Dual pixels give a Dual direction, so the bearing can be differentiated with respect to the pixel; integer pixels give the bearing of their floating-point value, as the lens-level backproject(lens, xd) and the grid models' backproject do.

The return type and semantics depend on the intrinsics type:

  • LogicalIntrinsics: Returns normalized unitless direction

Arguments

  • model::CameraModel: Camera model with intrinsics and lens
  • u::StaticVector{2,T}: 2D pixel coordinates [u, v] (unitless, in px)

Returns

  • Vec3: Normalized ray direction (unitless)

Example

camera = CameraModel(16.0mm, Size2(width=5.86μm/px, height=5.86μm/px), [320.0px, 240.0px])
u = SVector(400.0, 300.0)  # px (unitless)
ray = backproject(camera, u)  # Returns normalized Vec3 direction
source
VisualGeometryToolkitCore.project — Method
project(model::CameraModel, X::AbstractVector) -> Point2

Project 3D point X in camera coordinates to 2D pixel coordinates.

Arguments

  • model::CameraModel: Camera model
  • X::AbstractVector: 3D point in camera coordinates [X, Y, Z] (with units)

Returns

  • Point2 with pixel coordinates (including px units)

Example

model = CameraModel(16.0mm, Size2(width=5.86μm/px, height=5.86μm/px), [320.0px, 240.0px])
X = SVector(10.0mm, 5.0mm, 100.0mm)
u = project(model, X)  # Returns Point2 in pixels
source
VisualGeometryToolkitCore.unproject — Method
unproject(model::CameraModel{<:LogicalIntrinsics}, u::AbstractVector, depth::Real) -> Point3
unproject(intrinsics::LogicalIntrinsics, lens, u, depth) -> Point3

The camera-frame point on the ray behind pixel u whose Z-DEPTH is depth — the convention of depth images: distance along the optical axis. It is b·(depth / b₃) with b = backproject(intrinsics, lens, u), for every lens, and NaN where the ray has b₃ ≤ 0: a Z-depth names no point on such a ray. A point at distance d ALONG any ray, past 90° included, is d · backproject(model, u). A projective camera has no metric ray, so no Z-depth: there is no method, and the call raises a MethodError.

source
VisualGeometryToolkitCore.Cameras.angle_of_view — Method
angle_of_view(camera::BoundedCamera; dimension = :horizontal) -> θ

The angle of view of a bounded camera: the FULL angle, in radians, of the scene it IMAGES in dimension — its pixel extent, bounded by its lens_coverage (where the lens covers the sensor) — read through the camera's own backproject (K⁻¹, then the lens), so a distorted or fisheye lens answers its true angle and an anisotropic K uses the focal length of that axis. For a pinhole with a centred principal point it is the inverse of focal_length(resolution, θ; dimension), which takes the same full angle (focal_length(sensor, θ) is its metric form). (The lens's own limit, view_angle_limit, is a HALF angle from the optical axis.) The size this angle spans at a distance is field_of_view.

The sample points, each moved in along the line from the principal point to where the lens coverage ends when it lies past it (a circular fisheye's dark corners, the 14 mm disc cut by a wide sensor):

  • :horizontal: the two edge points on the principal point's row, (0, c_y) and (width, c_y);
  • :vertical: the two on its column, (c_x, 0) and (c_x, height);
  • :diagonal: the four corners of the extent.

The answer is twice the SMALLEST of those points' view angles — the nearer edge when the principal point is off-centre, so the cone of that full angle about the optical axis is imaged in that dimension. A camera whose whole sensor is covered (the default coverage) reads the sensor's edges. A principal point outside the coverage images nothing about the axis: an ArgumentError. For a lens that is not radially symmetric the valid_domain is the star polygon through each azimuth's fold, pulled in by at most its chords' sag (R·10⁻⁵), so the answer can understate the angle by that much.

A SplineLens camera reads about its gauge principal point, the pixel imaging ẑ, so the angles are about the optical axis.

Clipping by the model: a point outside the lens model's valid_domain (or one the lens does not image) counts as the lens's own limit, so when the valid domain clips the covered area in that dimension the answer is 2 · view_angle_limit(lens).

cam = BoundedCamera(CameraModel(K; lens = Lens(Equidistant())), sensor("Sony", "IMX250"))
angle_of_view(cam)                          # horizontal, full angle in radians
angle_of_view(cam; dimension = :diagonal)
source
VisualGeometryToolkitCore.Cameras.field_of_view — Method
field_of_view(camera::BoundedCamera, distance; dimension = :horizontal) -> size

The field of view of a bounded camera at distance: the SIZE of the scene it images there, on a plane perpendicular to its optical axis — 2·distance·tan(θ/2) for its angle_of_viewθ in dimension, in the unit of distance (a length such as 2.0m, or a plain number). An angle of view of 180° or more spans no finite size on a plane, and a distance must be positive: both an ArgumentError. (Not the field-of-view LENS model FieldOfView, Devernay's distortion — a different thing that shares the words.)

field_of_view(cam, 3.0m)                         # the width of the scene imaged 3 m away
field_of_view(cam, 3.0m; dimension = :vertical)
source
VisualGeometryToolkitCore.Cameras.lookat — Method
lookat(eye::StaticVector{3, <:Real}, target::StaticVector{3, <:Real}, up::StaticVector{3, <:Real};
       convention::Symbol = :cv)

Create a camera extrinsics transform that looks from eye position toward target.

Unitless version for internal use and backwards compatibility.

convention selects the camera-frame convention (same symbol-selector precedent as Rect(sensor, :julia)):

  • :cv (default) — Computer Vision convention:
    • +Z axis points FORWARD (toward target)
    • +X axis points RIGHT
    • +Y axis points DOWN
    • Camera -Y aligns with world up direction (supply up as [0, -1, 0])
  • :gfx — graphics convention (Mitsuba/OpenGL-style look_at frame): camera-to-world columns z = normalize(target − eye), x = normalize(cross(up, z)), y = cross(z, x); supply up as the world up direction (typically [0, 1, 0]). The returned map is still the world-to-camera extrinsics — only the frame construction differs.

Arguments

  • eye::StaticVector{3}: Camera position in world coordinates (plain numbers, in the caller's unit)
  • target::StaticVector{3}: Point to look at in world coordinates (same unit)
  • up::StaticVector{3}: World up direction (see per-convention notes above)

Returns

  • CameraExtrinsics: World-to-camera transformation (frame-labeled EuclideanMap)

Example

# Camera at (0, 0, 0) looking toward (0, 0, 1000), with world -Y pointing up
extrinsics = lookat(SVector(0.0, 0.0, 0.0), SVector(0.0, 0.0, 1000.0), SVector(0.0, -1.0, 0.0))

See also lookat_pose for the camera-to-world form (bit-exact eye translation, no inversion round-trip).

source
VisualGeometryToolkitCore.Cameras.lookat — Method
lookat(eye::StaticVector{3, <:Quantity}, target::StaticVector{3, <:Quantity}, up::StaticVector{3, <:Real};
       convention::Symbol = :cv)

Create a camera extrinsics transform that looks from eye position toward target.

A PHYSICAL pose: eye and target are lengths, and the extrinsics' translation is a length in metres — the plain lookat of the two points in metres, read back in metres (a plain point beside a length is a DimensionError).

Uses Computer Vision convention:

  • +Z axis points FORWARD (toward target)
  • +X axis points RIGHT
  • +Y axis points DOWN
  • Camera -Y aligns with world up direction

Arguments

  • eye::StaticVector{3, <:Quantity}: Camera position in world coordinates (physical lengths)
  • target::StaticVector{3, <:Quantity}: Point to look at in world coordinates (physical lengths)
  • up::StaticVector{3, <:Real}: World up direction, unitless (typically [0, -1, 0] for CV convention where -Y is up)

Returns

  • CameraExtrinsics: World-to-camera transformation (frame-labeled EuclideanMap, physical)

Example

# Camera at (0, 0, 0) looking toward (0, 0, 1000mm), with world -Y pointing up
extrinsics = lookat(SVector(0.0mm, 0.0mm, 0.0mm), SVector(0.0mm, 0.0mm, 1000.0mm), SVector(0.0, -1.0, 0.0))
source
VisualGeometryToolkitCore.Cameras.lookat_pose — Method
lookat_pose(eye::StaticVector{3, <:Real}, target::StaticVector{3, <:Real}, up::StaticVector{3, <:Real};
            convention::Symbol = :cv) -> CameraPose

Camera POSE (camera-to-world transform) that looks from eye toward target, as a typed CameraPose — the pose-side companion of lookat, sharing its convention selector (:cv / :gfx).

Built directly from the frame columns (rotation = camera basis as camera-to-world columns, translation = eyebit-exactly) rather than as inv(lookat(...)), so no inversion round-trip touches the translation. Feed the result straight to the frame-typed camera constructor PosedCamera(model, lookat_pose(...)), which inverts internally into the extrinsics slot.

source
VisualGeometryToolkitCore.focal_length — Method
focal_length(sensor::Sensor, θ::Real; dimension=:horizontal)

The physical focal length from the FULL field of view angle and the sensor: focal_length(sensor.resolution, θ) pixels times the pitch of that axis.

Arguments

  • sensor::Sensor: the sensor (its resolution and pitch)
  • θ::Real: the full angle of view in radians
  • dimension::Symbol: Which dimension the angle of view applies to (:horizontal or :vertical)

Returns

  • Physical focal length (a length quantity)

Examples

s = sensor("Sony", "IMX174")
f_mm = focal_length(s, deg2rad(40.0))
source
VisualGeometryToolkitCore.focal_length — Method
focal_length(resolution::Size2, θ::Real; dimension=:horizontal)

The focal length an angle of view implies: in pixels from the FULL angle of view (plain radians — angles carry no dimension in the VGC unit system) and resolution. Its inverse, for a camera, is angle_of_view. The other forms: focal_length(sensor::Sensor, θ) (a length) and the accessor focal_length(intrinsics) ((fx, fy) in pixels).

Arguments

  • resolution::Size2: Image resolution (width, height) in pixels: plain numbers or dimensionless quantities (px); a length is a DimensionError
  • θ::Real: the full angle of view in radians
  • dimension::Symbol: Which dimension the angle of view applies to (:horizontal or :vertical)

Returns

  • Focal length in pixels (same logical unit as resolution)

Examples

focal_length(Size2(width=1280px, height=960px), 0.698)
focal_length(Size2(width=1280px, height=960px), deg2rad(40.0))
source
VisualGeometryToolkitCore.p3p — Method
p3p(model::CameraModel, correspondences) -> FixedModels{4, Extrinsics3}

Solve the Perspective-3-Point (P3P) problem using correspondences.

This is a thin wrapper around the core p3p(rays, X) solver that accepts correspondences directly. It extracts the 3D world points and 2D image points from the correspondences, backprojects the image points to rays, and calls the core solver.

Arguments

  • model::CameraModel: Camera model for backprojecting image points to rays
  • correspondences: Correspondence container with exactly 3 point pairs (3D world → 2D image); any container satisfying the sources/targets protocol (Correspondences, StructArray of Pairs, plain Pair vector)

Returns

  • FixedModels{4, Extrinsics3}: Stack-allocated container of 0–4 rigid poses. Empty (length == 0) if no valid solution exists. Iterate with for ext in result ... end or collect(result).

Examples

# Setup camera
s = sensor("Sony", "IMX174")
sensor_bounds = Rect(s)
f = focal_length(s, deg2rad(40.0); dimension=:horizontal)
pp = centroid(sensor_bounds) .* px
model = CameraModel(f, s.pitch, pp)

# Create correspondences (3D world points → 2D image points)
X3 = [Point3(100.0, 100.0, 0.0), Point3(200.0, 150.0, 0.0), Point3(150.0, 200.0, 0.0)]
u3 = [Point2(640.0, 480.0), Point2(700.0, 500.0), Point2(680.0, 550.0)]
cs = correspondences(X3, u3)

# Solve P3P — always a FixedModels; empty (length 0) when no valid pose
result = p3p(model, cs)
cameras = PosedCamera.(Ref(model), collect(result))

See Also

  • p3p(rays, X): Core P3P solver that works with rays directly
  • sample_p3p: Sample valid camera poses by randomly sampling correspondences
source
VisualGeometryToolkitCore.Cameras.sample_p3p — Method
sample_p3p(model, X3::SVector{3}, sensor_bounds, plane::Plane3D; max_retries=1000)

Sample a P3P configuration using 3 fixed world points with winding-consistent image sampling.

Given 3 world points (e.g., board corners), samples 3 random image points and sorts them with consistent winding (world CCW, image CW due to y-axis flip). This ensures:

  • Deterministic scale: World point distances define the scale
  • Consistent orientation: Matching winding prevents reflected/flipped solutions

Points can be supplied in any order - the function sorts world points to CCW and image points to CW (matching winding due to y-axis inversion between coordinate systems).

Arguments

  • model::CameraModel: Camera model
  • X3::SVector{3,Point3}: 3 world points (any order, will be sorted to CCW)
  • sensor_bounds::Rect: Sensor bounds for sampling image points
  • plane::Plane3D: Reference plane for chirality test

Returns

  • (camera, correspondences): Camera and the 3 correspondences used (in CCW order)

Example

# Board corners in world coords (centered, y-up, z=0)
board_rect = Rect(bundle, :world)
corners = coordinates(board_rect)  # 4 corners
X3 = SVector{3}(Point3.(vcat.(corners[[1,2,3]], 0.0)))  # any 3 corners

camera, correspondences = sample_p3p(model, X3, sensor_bounds, Plane3D())
source
VisualGeometryToolkitCore.Cameras.sample_p3p — Method
sample_p3p(model, X, sensor_bounds, plane::Plane3D; max_retries=1000)
sample_p3p(rng, model, X, sensor_bounds, plane::Plane3D; max_retries=1000)

Sample a random P3P configuration with chirality filtering.

Like sample_p3p(model, X, sensor_bounds) but additionally filters solutions to only those where the camera is on the positive side of the given plane (i.e., signed_distance(plane, camera_center) > 0).

This is useful for planar calibration targets where you want to ensure the camera is "in front of" the target plane.

Arguments

  • rng=Random.default_rng(): Random number generator (optional first positional argument)
  • model::CameraModel: Camera model
  • X::Vector{<:StaticVector{3,Float64}}: 3D world points on or near the plane
  • sensor_bounds::Rect: Sensor bounds for sampling image points (unitless, in pixels)
  • plane::Plane3D: Reference plane for chirality test

Keyword Arguments

  • max_retries=1000: Maximum number of sampling attempts

Returns

  • (camera, correspondences): Tuple containing:
    • camera::PosedCamera: The camera with valid pose (in front of plane)
    • correspondences::StructArray{Pair{Point3, Point2}, 1, ...}: The 3 correspondences used

Example

# World coordinate system: origin at board center, z=0 plane, z-axis pointing out
board_plane = Plane3D()  # z=0, normal = (0,0,1)

# Sample camera that is in front of the board (z > 0)
camera, correspondences = sample_p3p(model, X, sensor_bounds, board_plane)
source
VisualGeometryToolkitCore.Cameras.sample_p3p — Method
sample_p3p(model, X, sensor_bounds; max_retries=1000)
sample_p3p(rng, model, X, sensor_bounds; max_retries=1000)

Sample a random P3P configuration by sampling from world points X and random image points.

Creates 3D-2D correspondences by:

  1. Sampling 3 random points from X (world coordinates)
  2. Sampling 3 random 2D image points in sensor bounds
  3. Pairing them as correspondences and solving P3P

Takes a logical camera (CameraModel{<:LogicalIntrinsics}): X and the sensor bounds are unitless (the construction boundary stripped units).

Arguments

  • rng=Random.default_rng(): Random number generator (optional first positional argument)
  • model::CameraModel: Camera model
  • X::Vector{<:StaticVector{3,Float64}}: 3D world points, unitless
  • sensor_bounds::Rect: Sensor bounds for sampling image points (unitless, in pixels)

Keyword Arguments

  • max_retries=1000: Maximum number of sampling attempts

Returns

  • (extrinsics, correspondences): Tuple containing:
    • extrinsics::Vector{Extrinsics3}: Camera extrinsics (world-to-camera transforms) for valid P3P solutions
    • correspondences::StructArray{Pair{Point3, Point2}, 1, ...}: The 3 correspondences used for P3P, backed by SVector{3} columns

Throws an error if no valid configuration is found after max_retries attempts.

Examples

# Setup camera
s = sensor("Sony", "IMX174")
sensor_bounds = Rect(s)
f = focal_length(s, deg2rad(40.0); dimension=:horizontal)
pp = centroid(sensor_bounds) .* px
model = CameraModel(f, s.pitch, pp)

# 3D points (planar points at z=0)
X = [Point3(100.0, 100.0, 0.0), Point3(200.0, 150.0, 0.0), ...]

# With default RNG
extrinsics, correspondences = sample_p3p(model, X, sensor_bounds)

# With explicit RNG for reproducibility
rng = Random.MersenneTwister(12345)
extrinsics, correspondences = sample_p3p(rng, model, X, sensor_bounds)

# Access correspondence data (StructArray interface)
X_used = correspondences.first   # All 3D points
u_used = correspondences.second  # All 2D image points
# Or index individual correspondences: correspondences[1].first, correspondences[1].second

# Use best extrinsics (e.g., based on reprojection error)
cameras = PosedCamera.(Ref(model), extrinsics)
best_camera = cameras[1]  # Or select based on reprojection error
source

Intrinsics and Sensors

The intrinsics and sensor types are on the Intrinsics and Sensors API page.

Boards and canvases

A planar target's Board is the sensor's analogue: the printed board's size (lengths, or plain numbers for a unitless target) and its corner in the design chart; Rect(board) is its domain there, as Rect(sensor) is the image's in pixels. The Canvas is the printed page the board lies on, with the image printed on it (CanvasTexture: a relative path, its SHA-256 and the rectangle it covers) — canvas ⊃ board ⊃ pattern, all rectangles of the one design chart the Canvas docstring states.

VisualGeometryToolkitCore.Cameras.Board — Type
Board{T}

The calibration board a planar target's pattern is printed on, the target's analogue of a Sensor: its size and where it lies in the design chart. It follows the target's unit rule: physical (lengths, stored in metres) for a physical target, plain numbers for a unitless one.

Fields

  • size::Size2{T}: the board proper, width × height — BlobBoards' board_size, the printed board inside the sheet's white margin, frame and corners included; NOT its canvas_size (the sheet) and not the pattern's extent. Lengths (a quantity of any other dimension is a DimensionError) or plain numbers; a mix is an ArgumentError.
  • origin::SVector{2, Float64}: the board's lower-left corner in the design chart (the coordinates of the nominal points, stated once at Canvas): plain metres for a physical board, the design's plain numbers for a unitless one.

Read them as properties, board.size and board.origin, as a sensor's resolution and pitch are.

Constructors

Board(Size2(width = 297mm, height = 210mm))                              # corner at the design origin
Board(Size2(width = 297mm, height = 210mm), [-10.0mm, -10.0mm])          # the corner as lengths
Board(Size2(width = 297mm, height = 210mm), SVector(-0.01, -0.01))       # or plain metres
Board(Size2(width = 297.0, height = 210.0), [-10.0, -10.0])              # a unitless board

Rect(board) is the board's domain in the design chart in the board's own numbers (plain metres when physical) — what a target's surface is bounded by (grown by a 16-eps rounding margin), as Rect(sensor) is the image's domain in pixels. ustrip(u, board) and with_unit(board, u) move it between the two forms, as they move a target's points.

See Also

source
VisualGeometryToolkitCore.Cameras.Canvas — Type
Canvas{T}

The page a planar target is printed on: the sheet that holds the Board (the calibrated rectangle) and around it the margin; the pattern is what is printed on the board. A canvas ⊃ board ⊃ pattern hierarchy, the canvas being the target's record of what was printed, not of what was calibrated.

The design chart

The board, the canvas and a canvas texture are axis-aligned rectangles of ONE chart, the design chart: the plane coordinates of a planar target's nominal points, x to the right and y up as the printed side is seen from the front, in metres for a physical target (lengths at the constructors, plain metres stored) and in the design's plain numbers for a unitless one. Each rectangle is its lower-left corner (origin) and its size; the board's placement on the canvas is board.origin − canvas.origin.

Fields

  • size::Size2{T}: the page, width × height, by the board's length rule (lengths stored in metres, a quantity of any other dimension a DimensionError; or plain numbers; non-negative).
  • origin::SVector{2, Float64}: its lower-left corner in the design chart.
  • texture::Union{Nothing, CanvasTexture{T}}: the printed image (CanvasTexture), or nothing; in the canvas's form (FlexUnits' DimensionError otherwise).

Constructors

Canvas(Size2(width = 297mm, height = 210mm), [-20.0mm, -15.0mm])
Canvas(Size2(width = 297mm, height = 210mm), [-20.0mm, -15.0mm];
       texture = CanvasTexture("canvas.png", sha, Size2(width = 297mm, height = 210mm), [-20.0mm, -15.0mm]))
Canvas(Size2(width = 297.0, height = 210.0))                       # a unitless canvas

Rect(canvas) is its rectangle in its own numbers. ustrip(u, canvas) and with_unit(canvas, u) move it (and its texture) between the two forms, as they move a board. A PlanarTarget carries it beside its board (canvas = …; canvas(t) reads it).

source
VisualGeometryToolkitCore.Cameras.CanvasTexture — Type
CanvasTexture{T}

The image printed on a Canvas: a reference to the image file and the rectangle of the design chart it covers. VGC never reads the image; the reference is what a reader resolves and checks.

Fields

  • path::String: the image file, RELATIVE to the file that names it (a rig or board file); an absolute path is an ArgumentError.
  • sha256::String: the SHA-256 of the image file's bytes, 64 lowercase hex digits (an ArgumentError otherwise) — the reader's integrity check.
  • size::Size2{T}, origin::SVector{2, Float64}: the rectangle the image covers, by the board's rule (lengths stored in metres or plain numbers; its lower-left corner in the design chart, plain metres when physical). The size is positive.

The image's first row lies along the rectangle's top edge (largest y) and its first column along the left edge (smallest x); its outer pixel edges lie on the rectangle's edges, so each pixel of a W × H image covers width/W × height/H of the chart.

Constructors

CanvasTexture("board.png", sha, Size2(width = 297mm, height = 210mm))                    # corner at the design origin
CanvasTexture("board.png", sha, Size2(width = 297mm, height = 210mm), [-10.0mm, -5.0mm])
CanvasTexture("board.png", sha, Size2(width = 297.0, height = 210.0), [-10.0, -5.0])     # a unitless one

Rect(texture) is the covered rectangle in its own numbers (plain metres when physical).

source

The overlap of a rotating camera's views

view_overlap measures, for every pair of frames of a camera turning about its centre, the solid angle of the directions both image and its share of each frame's footprint: the overlap a panorama's pair selection reads.

VisualGeometryToolkitCore.Cameras.view_overlap — Method
view_overlap(cameras; boundary_points = 512) -> StructVector
view_overlap(a, b; boundary_points = 512) -> NamedTuple

The directions two frames of a camera turning about its centre both image, for every pair i < j of cameras: one row (; i, j, solid_angle, fraction_i, fraction_j) per pair, K(K − 1)/2 rows in the order (1, 2), (1, 3), …, (2, 3), … as a StructVector, so rows.solid_angle is a column. solid_angle is the solid angle (sr) of the intersection of the two frames' footprints on the sphere; fraction_i is it over frame i's own footprint, and fraction_j over frame j's. A pair of cameras a, b gives the one row, i = 1, j = 2. For a panorama's pair selection: a pair worth matching shares a fraction of its view.

Each camera is a BoundedCamera of an OrientedCamera, a frame posed by its orientation alone, BoundedCamera(PosedCamera(model, relabel(CameraFrame, WorldFrame, RotationMap(R_k))), extent) for a camera → world rotation R_k, d_world = R_k · d_camera, whose model is a CameraModel (another model, a grid one, is an ArgumentError). Frames may have different cameras. A camera posed by an EuclideanMap has a position, and an unposed one no orientation: both are an ArgumentError naming the fix. Frame k's footprint is the set of directions R_k · backproject(model, u) over its region Ωk, [`imagedregion](@ref)(camera): the sensor ∩ the camera's lens coverage (its image circle, [lenscoverage](@ref)) ∩ where the lens model is valid ([validdomain`](@ref)). Each frame carries its own lens coverage; a camera whose whole sensor is lit gives the extent where the lens model holds, which is not where light falls for a lens whose image circle lies inside the sensor.

The pair (i, j) is computed once, in frame i's image:

  1. Ωi is a ring of `boundarypoints` pixels along its boundary: the extent's edges (every corner exact), the circle's samples, and their convex intersection by Core's Sutherland–Hodgman.
  2. Pj is frame j's ring drawn into image i: each pixel back-projected by j, rotated by `Riᵀ Rjand projected byi, each edge split into geodesic pieces no longer in imagei` than Ωi's own edges. A direction frame i does not image (behind a perspective camera, past a lens's fold, NaN, or one whose pixel does not back-project to it within 10⁻⁶ rad) has no pixel; the ring is cut there and continued along a far circle about i's principal point, four times Ωi's radius, at the direction's azimuth, so it stays a closed curve that runs outside Ωi wherever i cannot draw it.
  3. Sutherland–Hodgman clips Pj by the convex Ωi. A curve of any shape keeps its winding number inside the clip, so the clipped ring's area is the integral of Pj's winding number over Ωi; pieces a non-convex Pj splits into come back joined by edges along Ωi's boundary, traversed once each way, which add nothing.
  4. That area is a solid angle: the clipped ring's vertices are back-projected by i to unit bearings and joined by geodesics, and their fan of signed spherical triangles from i's optical axis is summed (Van Oosterom & Strackee, tan(E/2) = a·(b × c) / (1 + a·b + b·c + c·a)). Along Ω_i's boundary the polygon follows the footprint's own geodesic edges: a vertex there is moved onto its edge's great circle, and a clipped edge along a sensor side gets the ring's vertices between its ends (a lens bends the side).
  5. The detours of item 2 are where the winding can be wrong, and only by an integer n constant over Ωi: a detour outside Ωi can only wind around all of it. One interior point of Ωi, the candidate farthest from Pj, fixes n: its winding number against whether frame j images its direction (project by j, in Ωj, back-projected within 10⁻⁶ rad). The overlap is the area less n times frame i's footprint. The same correction covers a footprint j that wraps frame i's backward direction (a wide fisheye behind a narrow lens), whose ring winds the other way round Ωi.

Each footprint's own solid angle is the same fan over its ring from its own optical axis, which needs no hemisphere: the sum is the enclosed area for any apex whose antipode is outside the region, and a footprint never reaches its camera's backward axis. So a lens past 180° (an 8 mm fisheye) is handled; a footprint whose ring comes within 0.05 rad (≈ 3°) of its camera's backward axis is an ArgumentError, as is a region whose boundary the lens cannot back-project and a principal point outside its region.

A perspective camera maps geodesics to lines, so its footprints are exact great-circle polygons and their overlap is exact at any boundary_points. A curved boundary (an image circle, a distorted side) is a polygon of boundary_points vertices, so the error is the chords' sag, O(1/boundary_points²): 1.6 × 10⁻⁵ sr of a 0.77 sr cap at 512 (2 × 10⁻⁵ relative), a quarter of it at 1024. A footprint that coincides with another (the same camera and rotation) clips to itself: fraction 1. A pair whose footprints' bounding caps about the optical axes are disjoint is 0 without clipping.

cam = CameraModel(K; lens = KannalaBrandt(-0.135))
bare(R) = BoundedCamera(PosedCamera(cam, relabel(CameraFrame, WorldFrame, RotationMap(R))), sensor("Sony", "IMX250"))
frame(R) = (b = bare(R); with_lens_coverage(b, image_circle(b, 6.0mm)))   # its lens coverage: the datasheet's 6 mm circle
frames = [frame(RotY(2π * k / 11)) for k in 0:10]   # 11 frames about the vertical axis
rows = view_overlap(frames)
rows[1].fraction_i                                # the share of frame 1's view frame 2 sees
source

Index