Cameras API
Cameras, camera models, intrinsics, and sensors. The grid cameras (a SplineLens, a NoncentralSplineModel) are on the Spline Lens API page; calibration (the radial camera, intrinsics from homographies, view-angle limits) is on the Calibration API page; lens models and the lens contract with its batched surfaces are on the Lenses API page; the framed imaging chain, ray maps, and stereo rigs are on the Imaging Chain and Stereo API page. See the Cameras guide.
Cameras and Camera Models
VisualGeometryToolkitCore.Cameras — Module
CamerasCamera system submodule for VisualGeometryToolkitCore.
Provides complete camera modeling including:
- Sensors: Physical CMOS sensor specifications
- Intrinsics: Calibration matrices and intrinsic parameters (physical/logical)
- Lens Models: forward lenses (
Lens: a projection + a distortion in its frame) and lifts (LiftLens) - Camera Models: Composable camera models (intrinsics + lens)
- Cameras: Complete cameras in 3D space (model + pose)
- Stereo: Stereo camera rigs and epipolar geometry
- Utilities: Helper functions (focal_length, lookat, P3P solvers)
Main Types
PosedCamera,PosedPinholeCameraCameraModelLogicalIntrinsicsLens,LiftLens,Pinhole,KannalaBrandt,DivisionSensor,CMOS_SENSORSStereoRig
Main Functions
project,backproject,unprojectfocal_length,lookat,posep3p,sample_p3pdefault_frustum_depth
Example
using VisualGeometryToolkitCore
using VisualGeometryToolkitCore.Cameras
s = sensor("Sony", "IMX174")
camera = BoundedCamera(8mm, s; lens = Lens(DoubleSphere(0.25, 0.4))) # a lens on the sensor, centred
f = focal_length(s, deg2rad(40.0)) # or the focal length of a field of view
model = CameraModel(f, s.pitch, centroid(Rect(s)) .* px) # the general form: any principal point
posed = PosedCamera(model, lookat(...))VisualGeometryToolkitCore.Cameras.BoundedCamera — Type
BoundedCamera{C<:AbstractCamera, B, R} <: AbstractCameraA wrapper around any AbstractCamera that adds pixel-space image bounds (width × height) without disturbing the underlying camera's intrinsics or units. Used by SfM/MVS dataset readers (ETH3D, ScanNet, COLMAP exports) where the camera is described by its calibration plus an explicit image extent — information the bare PosedCamera type doesn't retain.
The wrapped camera may be UNPOSED — a bare model, is_posed(bc) == false — which is COLMAP's "camera" row (intrinsics, lens and image size, no pose) and the element of a stored camera collection (VisualGeometryToolkitIO's cameras kind). BoundedCamera(PosedCamera(model(bc), pose), bc.bound; lens_coverage = lens_coverage(bc)) is its posed counterpart; the pose accessors of an unposed one are an ArgumentError. is_posed is the wrapped camera's.
The bound
BoundedCamera(camera, extent::Size2) bounds by a pixel extent; BoundedCamera(camera, sensor::Sensor) by the physical sensor, whose resolution is the extent and whose pitch (a length per pixel) carries a lens datasheet's millimetres into pixels (image_circle). Either way extent(bc) is the Size2{Int} pixel extent; sensor(bc) is the Sensor, or nothing for the extent form. The bound is ONE field, bc.bound: the extent, or the sensor whose resolution is the extent.
The lens coverage
BoundedCamera(camera, bound; lens_coverage = region) — where light reaches the sensor, in pixels: a Circle or an Ellipse (the lens's image circle, measured with a detector or from a datasheet), or a Rect2; any real element type, converted to Float64, and kept as the kind given. Omitted, it is the sensor's own Rect: the whole sensor is lit. lens_coverage(bc) reads it, with_lens_coverage replaces it. What the camera images is imaged_region(bc): the sensor ∩ the lens coverage ∩ where the lens model is valid.
Field access
bc.camera,bc.bound,bc.lens_coverage— direct fields; read them throughmodel,extent/sensorandlens_coverage.
Method delegation
A BoundedCamera is THE type for pixel operations: every camera verb a CameraModel answers is forwarded to the wrapped camera (MacroTools.@forward, and @inlineproject/backproject for device kernels) — the maps and their Jacobians (project, backproject, backproject_with_jacobian, unproject, is_in_front, constraint_with_jacobians), the accessors (intrinsics, lens, pose, extrinsics, focal_length, model, calibration_matrix), the imaging and its domain (imaging_transform, imaging_chain, distortion_map, valid_domain, valid_rect, undistort, …) and the camera-derived solvers (p3p, transfer, radial_homography); so backproject_with_jacobian(bc, u) == backproject_with_jacobian(bc.camera, u). The bound applies to none of them (the extent is image_circle, imaged_region and lens_coverage). The model constructors (pinhole, window, with_parameters) and the parameter protocol are the model's: call them on model(bc), and rebound with with_model.
Trait
bounds_trait(::BoundedCamera) = HasBounds() — methods that need image extent dispatch on BoundsTrait: extent(bc) is the bound's pixel extent.
A device kernel's camera
The bound's kind and the region's are the type parameters B (Size2{Int}, or the stored Sensor type) and R (Rect2{Float64}, Circle{Float64} or Ellipse{Float64}), so a bounded camera is concrete, and isbits whenever the wrapped camera is (a CameraModel of an isbits lens): it travels into a GPU kernel as it is, its backproject/project the wrapped camera's. convert_eltype(Float32, bc) converts the wrapped camera and keeps the bound and the region.
Example
inner = PosedCamera(CameraModel(K; lens=Pinhole()), pose)
bc = BoundedCamera(inner, Size2(6048, 4032))
intrinsics(bc) # forwarded → CameraCalibrationMat
extent(bc) # Size2(6048, 4032)
lens_coverage(bc) # Rect2(0, 0, 6048, 4032): the whole sensor is litVisualGeometryToolkitCore.Cameras.BoundedCamera — Method
BoundedCamera(f::Quantity, sensor::Sensor; lens = Pinhole(), lens_coverage = nothing) -> BoundedCameraA lens of focal length f (a length, 8mm) on sensor: the camera model of focal length f / pitch pixels with its principal point at the sensor's centre (centroid(Rect(sensor))), bounded by the sensor, so its pitch stays with it (metric_focal_length). The camera of a datasheet, its lens model lens. A principal point off the centre is the general form, BoundedCamera(CameraModel(f, sensor.pitch, pp; lens), sensor).
sensor = Sensor(Size2(320, 240), 20μm)
camera = BoundedCamera(8mm, sensor; lens = Lens(DoubleSphere(0.25, 0.4))) # 400 px, principal point (160, 120)VisualGeometryToolkitCore.Cameras.OrientedCamera — Type
OrientedCamera{M}A camera that only turns about its centre: a PosedCamera whose extrinsics are a RotationMap{3}, PosedCamera{M, <:RotationMap{3}} (an alias, not a type of its own). It has an orientation and no position: extrinsics(c) is the world → camera rotation, pose(c) its inverse, and project, backproject and unproject act about the origin. Build it from the rotation of either direction: PosedCamera(model, relabel(CameraFrame, WorldFrame, RotationMap(R))) takes a camera → world orientation and inverts it, an unlabeled or world → camera RotationMap is stored as the extrinsics. What needs a position refuses one by name: a rig member, the plane-induced homography and the verbs on plane coordinates (project of a plane conic or rectangle, transfer_ellipse, HomographyMat, RadialCamera), and a non-central grid model's pose. The frames of a panorama are oriented cameras, as view_overlap takes them.
VisualGeometryToolkitCore.Cameras.PosedCamera — Type
PosedCamera{M, Rt} <: AbstractCameraA camera in 3D space, consisting of a camera model (intrinsics + lens) and extrinsics (pose).
Dispatch on the lens through the model parameter: PosedCamera{<:CameraModel{<:Any, <:MyLens}} (the PosedPinholeCamera alias is that form for Pinhole).
Use project(camera, X) to map a 3D world point to 2D pixel coordinates, and backproject(camera, u) / unproject(camera, u, depth) for the inverse directions. The transform is computed on demand and inlined.
Type parameters
M <: AbstractCamera: an UNPOSED camera model (CameraModel, or a grid model);PosedCamera(model, X)refuses a posed one (is_posed(model)) and aBoundedCamera— the bound goes outside the pose,BoundedCamera(PosedCamera(model, X), extent)Rt <: Union{EuclideanMap, RotationMap{3}}: extrinsics type. ARotationMapposes a camera that only turns about its centre, anOrientedCamera; a camera turns in three dimensions, so aRotationMap{2}is aMethodErrorat construction.
Fields
model::M: Camera model (intrinsics + lens)extrinsics::Rt: Camera extrinsics (world-to-camera transform)
Examples
# Create camera with logical intrinsics
model = CameraModel(800.0px, [320.0px, 240.0px])
extrinsics = EuclideanMap(RotMatrix(I), [0.0, 0.0, 0.0])
camera = PosedCamera(model, extrinsics)
# Dispatch on lens (via the model parameter; PosedPinholeCamera is the
# ready-made alias for the Pinhole case)
foo(::PosedPinholeCamera) = "pinhole path"
foo(::PosedCamera{<:CameraModel{<:AbstractIntrinsics, <:KannalaBrandt}}) = "fisheye path"
# Project a world point
X_world = SVector(1.0mm, 2.0mm, 10.0mm)
u = project(camera, X_world)VisualGeometryToolkitCore.Cameras.PosedCamera — Method
PosedCamera(model::CameraModel, e::CameraExtrinsics) -> PosedCamera
PosedCamera(model::CameraModel, p::CameraPose) -> PosedCameraConstruct a PosedCamera from a frame-labeled map, so the caller states which frame it holds instead of relying on the positional convention.
The bare positional constructor PosedCamera(model, extrinsics::EuclideanMap) with an UNLABELED map takes the world-to-camera map (documented convention, unchecked); a caller holding a camera-to-world map (a lookat-style pose, a renderer's sensor.to_world) who supplies it positionally stores the WRONG map without any error — every accessor then silently answers for the inverse camera. With frame labels that mistake is unrepresentable: CameraExtrinsics (being the extrinsics-labeled EuclideanMap) stores directly into the extrinsics slot via the default constructor, CameraPose inverts internally, and pose(PosedCamera(model, P::CameraPose)) ≈ P.
VisualGeometryToolkitCore.Cameras.PosedCamera — Method
PosedCamera(K::StaticMatrix{3,3}, R::StaticMatrix{3,3}, t::StaticVector{3};
check=true, project=false) -> PosedCameraBoundary constructor taking raw matrix data (typical for SfM dataset loaders). R is wrapped via the EuclideanMap(::SMatrix, ...) path which validates (and optionally snaps via project=true) before promoting to RotMatrix. t is plain numbers of K's element type, or lengths (t .* m), which give a physical camera.
VisualGeometryToolkitCore.Cameras.PosedPinholeCamera — Type
PosedPinholeCameraCameras whose lens is exactly Pinhole — the alias for lens dispatch now that PosedCamera carries only {M, Rt} (the former {I, L, Rt, M} repeated what M already determines). Consumed by the induced-homography constructors in geometry/cameras/homography.jl.
VisualGeometryToolkitCore.Cameras.calibration_matrix — Method
calibration_matrix(camera::PosedCamera) -> SMatrix{3,3,Float64,9}Return the 3×3 calibration matrix K as a plain SMatrix. For a SplineLens camera K is the lens's gauge — the pinhole at the pixel imaging ẑ (skew 0) — kept as it is by every fit.
Example
camera = PosedCamera(CameraModel(; focal=800.0, pp=(320.0, 240.0)), ext)
K = calibration_matrix(camera) # [800 0 320; 0 800 240; 0 0 1]VisualGeometryToolkitCore.Cameras.default_frustum_depth — Method
default_frustum_depth(camera::PosedCamera) -> Float64Default frustum depth for visualization: fx from K, a natural depth scale in the camera's working units.
VisualGeometryToolkitCore.Cameras.is_in_front — Method
is_in_front(camera, x, y, z) -> AbstractArray{Bool}Broadcasted component-array (SoA) form of the chirality test: world-frame point components in, camera-frame z > 0 mask out. Per-element semantics match is_in_front(camera, Point3(xᵢ, yᵢ, zᵢ)); the component layout keeps the whole test one fused broadcast on GPU arrays (hoisted from the RayTracedCalibration GPU pipeline).
VisualGeometryToolkitCore.Cameras.is_in_front — Method
is_in_front(camera, point::Point3) -> BoolChirality test: returns true if point is in front of the camera (z > 0 in camera coords).
Broadcasts over points:
mask = is_in_front.(Ref(camera), world_points)VisualGeometryToolkitCore.Cameras.lens_coverage — Method
lens_coverage(camera::BoundedCamera, rim_margin::Real) -> Union{Rect2{Float64}, Circle{Float64}, Ellipse{Float64}, Nothing}The camera's lens_coverage less rim_margin pixels at the LENS's edge (an image circle's soft rim), or nothing when nothing is left: a Circle of radius r − rim_margin, an Ellipse with both semi-axes less rim_margin, a Rect inset by rim_margin on each side that lies inside the sensor — a side on or past the sensor's edge is the sensor's, not the lens's, and stays, so a camera whose whole sensor is lit loses nothing. The kind it was given; a margin of 0 returns the coverage. Where a consumer counts a pixel as lit away from the rim: imaged_region(with_lens_coverage(camera, c)) is what such a camera images (with with_lens_coverage), and a Circle's c.r the radius it reaches from the circle's centre.
VisualGeometryToolkitCore.Cameras.lens_coverage — Method
lens_coverage(camera::BoundedCamera) -> Union{Rect2{Float64}, Circle{Float64}, Ellipse{Float64}}Where light reaches the camera's sensor, in its pixels: the lens's image circle (a Circle or an Ellipse, measured or from a datasheet), or the sensor's own Rect when the whole sensor is lit — the kind it was given, in Float64. Not where the lens MODEL is valid — that is valid_domain; what the camera images is both, inside the sensor: imaged_region(camera).
VisualGeometryToolkitCore.Cameras.metric_focal_length — Method
metric_focal_length(camera::BoundedCamera) -> lengthThe camera's focal length as a length: its fx pixels times the pixel pitch of the sensor it is bounded by (fx · pitch.width, the lens's focal length when the camera is a lens on that sensor, BoundedCamera(f, sensor)). A camera bounded by a bare pixel extent has no pitch: an ArgumentError.
VisualGeometryToolkitCore.Cameras.plane_map — Method
plane_map(camera::PosedCamera) -> CoordinateTransformations.TransformationThe board-plane → pixel map of a posed camera, for every camera model: the z = 0 plane of its world (a planar target's design plane, in its units) lifted to 3-D (q ↦ (q, 0), the target surfaces' plane lift), the camera's rigid extrinsics, then its model's projection, project(camera.model, ·) as a map (the forward twin of Backprojection). plane_map(camera)(q) is project(camera, [q; 0]), and transform_deriv(plane_map(camera), q) its 2 × 2 Jacobian with respect to the plane point, the composition's chain rule (the projection's by ForwardDiff through project, a spline model's by the implicit function theorem).
For a central CameraModel the same map, on the plane points in front of the camera, is the plane-induced homography of its pinhole twin, dehomogenised, then its lens's distortion_map composition (a pinhole: the homography map alone, PerspectiveMap() ∘ HomographyMap(camera)), and plane_map returns that form: it is the composition transfer_ellipse(camera, shape) transfers a plane conic through, stage by stage, so a conic's image is the homography's exact conic action, then the lens's local affinity. A plane point behind the camera (Z ≤ 0) has no image (project is NaN or a ray past 90°), and the homography form does not test it. A SplineLens camera whose lattice does not image ẑ (the off-axis gauge, which distortion_map refuses for an undistortion) takes the same form: the gauge's K cancels between the homography and the lens stage, so the map is the lens's own projection of the plane point.
The camera's extrinsics are plain numbers in the plane's unit: a physical camera's plain form is ustrip(m, camera).
VisualGeometryToolkitCore.Cameras.sensor — Method
sensor(camera::BoundedCamera) -> Union{Sensor, Nothing}The physical Sensor a camera is bounded by — its resolution is the camera's extent, its pitch the length per pixel — or nothing for a camera bounded by a bare pixel extent. The one-argument accessor beside the two-argument database lookup sensor(manufacturer, model): both answer a Sensor.
VisualGeometryToolkitCore.Cameras.with_directions — Method
with_directions(camera::BoundedCamera, D::AbstractMatrix{<:StaticVector{3}}) -> BoundedCameraA bounded spline camera (a SplineLens camera or a NoncentralSplineModel, posed or not) with the direction controls D: its model's with_directions, the bound, the lens coverage and a pose kept.
VisualGeometryToolkitCore.Cameras.with_lens_coverage — Method
with_lens_coverage(camera::BoundedCamera, region) -> BoundedCameraThe same camera and bound with the lens coverage region (a Rect2, Circle or Ellipse of any real element type, converted to Float64; nothing for the whole sensor).
VisualGeometryToolkitCore.Cameras.with_present — Method
with_present(camera::BoundedCamera, present::AbstractMatrix{Bool}) -> BoundedCameraA bounded spline camera (a SplineLens camera or a NoncentralSplineModel, posed or not) holding the nodes present: its model's with_present, the bound, the lens coverage and a pose kept.
VisualGeometryToolkitCore.backproject — Method
backproject(camera::PosedCamera, u::StaticVector{2, Float64}) -> Vec3Backproject a 2D pixel to a 3D ray direction in world coordinates (normalized). Returned as a Vec3 because it represents a direction — not translatable under affine maps. The ray's origin is the camera center (pose(camera).t); points along it are pose(camera).t + s · dir for s > 0.
VisualGeometryToolkitCore.extent — Method
extent(cam) -> Size2{Int}Pixel-space image extent (width × height). Trait-dispatched: errors at compile time for cameras that don't carry bounds.
VisualGeometryToolkitCore.extrinsics — Method
extrinsics(camera) -> CameraExtrinsicsCamera extrinsics (world-to-camera transform). Returns a frame-labeled map (WorldFrame → CameraFrame): the extrinsics slot is world-to-camera by contract, so the accessor labels it even when the camera was constructed from an unlabeled map (identity when already labeled).
camera = PosedCamera(model, ext)
E = extrinsics(camera) # World-to-camera
E(world_point) # Transform point to camera coordinatesVisualGeometryToolkitCore.focal_length — Method
focal_length(camera::PosedCamera)Get focal length from camera's intrinsics: the (fx, fy) tuple in pixels.
Examples
K = CameraCalibrationMat(800.0px, [320.0px, 240.0px])
model = CameraModel(LogicalIntrinsics(K), Pinhole())
camera = PosedCamera(model, extrinsics)
fx, fy = focal_length(camera) # Returns (800.0, 800.0)VisualGeometryToolkitCore.model — Method
model(camera::PosedCamera) -> CameraModelReturn the camera's underlying CameraModel (intrinsics + lens), the single object the lower-level project/backproject/transfer kernels take. Method form so wrappers (e.g. BoundedCamera) can delegate via @forward. A rig's other member kind answers the same accessor: model(::PosedTarget) is its target.
VisualGeometryToolkitCore.pose — Method
pose(camera) -> CameraPoseCamera pose (camera-to-world transform), inverse of extrinsics. Returns a frame-labeled map (CameraFrame → WorldFrame).
camera = PosedCamera(model, extrinsics)
P = pose(camera) # Camera-to-world
P.t # Camera position in world coordinatesVisualGeometryToolkitCore.project — Function
project(camera::PosedCamera, rect::HyperRectangle{2}, n::Int=20) -> PolygonProject a 2D rectangle on the world z=0 plane, tessellating each edge into n samples.
VisualGeometryToolkitCore.project — Method
project(camera::PosedCamera, X_world::AbstractVector) -> Point2Project 3D point from world coordinates to pixel coordinates.
Arguments
camera::PosedCamera: Camera with model and extrinsicsX_world::AbstractVector: 3D point in world coordinates (with units)
Returns
- Point2 with pixel coordinates (px units)
Example
model = CameraModel(16.0mm, Size2(width=5.86μm/px, height=5.86μm/px), [320.0px, 240.0px])
extrinsics = lookat(SVector(0.0, 0.0, 10.0), SVector(0.0, 0.0, 0.0), SVector(0.0, 1.0, 0.0))
camera = PosedCamera(model, extrinsics)
X_world = SVector(1.0mm, 2.0mm, 3.0mm)
u = project(camera, X_world)VisualGeometryToolkitCore.project — Method
project(camera::PosedCamera, c::Circle3D) -> Ellipse or PolygonProject an arbitrarily-placed 3D circle through the camera. For pinhole cameras, the plane homography induced by the circle's local (u, v) frame gives the analytic image conic; other cases fall back to boundary sampling and return a Polygon.
VisualGeometryToolkitCore.project — Method
project(camera::PosedCamera, origin::Point3, dir::Vec3) -> (Point2, Vec2)Project a 3D origin and displacement through the camera and return the 2D arrow endpoints — the origin's image and the vector from the origin's image to (origin + dir)'s image. Use to plot a 3D normal or direction as an image-space arrow:
origin_2d, vec_2d = project(cam, c.center, c.radius * normal(c))
arrows2d!(ax, origin_2d, vec_2d; color=:magenta)VisualGeometryToolkitCore.project — Method
project(camera::PosedCamera, shape::Union{Circle, Ellipse}) -> Ellipse or PolygonProject a 2D circle or ellipse that lies on the world z=0 plane through the camera. For pinhole cameras, the analytic conic projection yields an Ellipse; partially-behind-camera cases or distorting lens models fall back to boundary-point sampling and return a Polygon.
VisualGeometryToolkitCore.unproject — Method
unproject(camera::PosedCamera, u::AbstractVector{Float64}, depth::Float64) -> Point3Unproject a 2D pixel at a given camera-space depth (Z along the camera's forward axis) to a 3D point in world coordinates. Returned as a Point3 because it represents a position — translatable under affine maps. Works for both PhysicalIntrinsics (metric units; depth in the chosen metric unit, e.g. mm) and LogicalIntrinsics (unitless pixels; depth carries whatever unit the caller picks for their scene).
VisualGeometryToolkitCore.Cameras.DEFAULT_MAX_FOV — Constant
DEFAULT_MAX_FOVRecommended half-angle cap for flattening a view onto a perspective plane, as the max_fov of undistorted_rect. 70° gives tan θ ≈ 2.75, so the rectified plane stays a small multiple of the frame; past ~80° (tan 5.7) it grows faster than the detail it reveals, and the last degree before 90° is unbounded.
A RECOMMENDATION, not the default: undistorted_rect defaults max_fov to Inf so that it and valid_rect stay bounded by the same rule. The callers that want a viewing cap — plot_rectification — supply this.
VisualGeometryToolkitCore.Cameras.CameraModel — Type
CameraModel{I, L} <: AbstractCameraCamera model combining intrinsics and lens model: an unposed camera (is_posed is false), the parametric intrinsics × lens family. The central grid camera is one of them, a SplineLens; the non-central grid, NoncentralSplineModel, is the other unposed camera.
Use project(model, X) to map a 3D camera-space point to 2D pixel coordinates, and backproject(model, u) / unproject(model, u, depth) for the inverse directions.
Fields
intrinsics::I: Camera intrinsics (LogicalIntrinsics)lens::L: Lens model (Pinhole, aLens, aLiftLens, …)
Examples
# Logical camera with pinhole lens
K = CameraCalibrationMat(800.0px, [320.0px, 240.0px])
logical = LogicalIntrinsics(K)
model = CameraModel(logical, Pinhole())
# Physical camera with pinhole lens
f = 16.0mm
pitch = Size2(width=5.86μm/px, height=5.86μm/px)
pp = [320.0px, 240.0px]
physical = LogicalIntrinsics(f, pitch, pp)
model = CameraModel(physical, Pinhole())
# With Brown-Conrady distortion
model = CameraModel(logical, Lens(Perspective(), Radial((-0.1, 0.01))))
# Project a point
X = SVector(10.0mm, 5.0mm, 100.0mm)
u = project(model, X)VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(map::AffineMap; lens=Pinhole()) -> CameraModelConstruct a logical camera model from a CoordinateTransformations.AffineMap.
The 2×2 linear part encodes focal lengths (and optional skew), the 2D translation encodes the principal point. This constructor enables idiomatic composition:
K = CameraCalibrationMat(M)
model = CameraModel(EuclideanMap(PixelFrame{:opencv}, PixelFrame{:colmap}) ∘ affine_part(K); lens=lens)VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(K::CameraCalibrationMat; lens=Pinhole(), pixel_frame=:colmap) -> CameraModelConvenience constructor from a calibration matrix K directly.
Wraps K in LogicalIntrinsics internally. The pixel_frame keyword specifies which pixel coordinate convention K is expressed in (:opencv, :colmap, or :julia). The principal point is shifted to COLMAP convention (pixel centers at 0.5) before storage.
Example
K = CameraCalibrationMat(800.0px, [320.0px, 240.0px])
model = CameraModel(K; lens=Lens(Perspective(), Radial((-0.1,))))
# From OpenCV calibration (pixel centers at 0,0)
K_cv = CameraCalibrationMat(M)
model = CameraModel(K_cv; lens=lens, pixel_frame=:opencv)VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(model::CameraModel, K′::CameraCalibrationMat) -> CameraModelThe same camera with intrinsics K′: the lens re-expressed under the new intrinsics matrix, so the pixel map is unchanged. A parametric lens shares the principal point: normalized′ = normalized / s with s = f′/f, so a scale-covariant radial lens transforms its coefficients by the law aᵢ → aᵢ·s^(2i), bⱼ → bⱼ·s^(2j) (the generalized k′ = k·s²); it throws for a moved principal point (the lens center is data, not a frame choice), for anisotropic rescaling of a radial lens (radial symmetry would be broken), and for families whose coefficients are only meaningful under a metric K. A SplineLens takes any skew-0 K′, a moved principal point included (its K is a gauge): its backward area moves by the change of the normalized plane, x′ = x / s + (c − c′)/f′, and so does a forward's output (its chart is about ẑ, which no K moves), so every projection is kept too.
VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(f::Quantity, pp::AbstractVector{<:Quantity}; lens=Pinhole()) -> CameraModelConvenience constructor to build a logical camera model from focal length and principal point (both in pixels).
Example
camera = CameraModel(800.0px, [320.0px, 240.0px])VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(f::Quantity, pitch::Size2{<:Quantity}, pp::AbstractVector{<:Quantity}; lens=Pinhole()) -> CameraModelConvenience constructor to build a complete physical camera model from parameters.
Example
camera = CameraModel(16.0mm, Size2(width=5.86μm/px, height=5.86μm/px), [320.0px, 240.0px])VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(K::SMatrix{3,3,<:Real}; lens=Pinhole(), pixel_frame=:colmap) -> CameraModelA camera model from a plain static 3×3 calibration matrix, as dataset readers and pool records store it: K is validated and wrapped as a CameraCalibrationMat, then built by the method above. With the default Pinhole() lens this is the pinhole camera of K. (A plain Matrix goes through CameraCalibrationMat(K) first.)
VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(; focal, pp, lens=Pinhole()) -> CameraModelUnitless convenience constructor for a logical camera model.
Accepts plain Real values (no px units required). Useful for synthetic data generation and quick prototyping.
Arguments
focal: Focal length in pixels — scalarRealfor isotropic, orTuple{Real,Real}for anisotropicpp::Tuple{Real,Real}: Principal point (cx, cy) in pixelslens::AbstractLensModel=Pinhole(): Lens model
Examples
model = CameraModel(; focal=800.0, pp=(320.0, 240.0))
model = CameraModel(; focal=(800.0, 805.0), pp=(320.0, 240.0)) # anisotropicVisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(m::RadialHomography{T, N}) -> CameraModel{ProjectiveIntrinsics{T}, Division{N, T}}The projective division camera of a radial homography: CameraModel(ProjectiveIntrinsics(center(m)), Division(division_coeffs(m)...)), built from m's stored primitives. Its ray frame is centred pixels, so the lens coefficients ARE the model's pixel-frame λ's; it drives the fused KA undistort! warp directly: undistort(CameraModel(m), img). No metric ray: a field-of-view operation (a finite max_fov of undistorted_rect) refuses it.
VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(f::Tuple{T,T}, pp::AbstractVector{<:Quantity}; lens=Pinhole()) where T<:Quantity -> CameraModelConvenience constructor to build a logical camera model with anisotropic focal lengths (fx ≠ fy).
Example
camera = CameraModel((800.0px, 805.0px), [320.0px, 240.0px])VisualGeometryToolkitCore.Cameras.backproject_with_jacobian — Method
See backproject(model, pixels).
VisualGeometryToolkitCore.Cameras.backproject_with_jacobian — Method
backproject_with_jacobian(model::CameraModel, u::StaticVector{2}) -> (f, M)The unit bearing of pixel u and its 3×2 Jacobian M = ∂f/∂u, in ONE evaluation. f is what backproject returns; M is what propagates a PIXEL covariance onto the sphere, so the pair is the entry point for calibrated two-view geometry on bearings (the tangent Sampson residual and its rows).
M is TANGENT by construction: fᵀM = 0 to rounding — a forward lens's is built in a tangent basis, and a backward lens's is the derivative of its unit bearing. Consumers therefore need no explicit projection onto the tangent plane — which is what keeps an inner loop free of the allocation such a projection would cost.
The tangency assertion is LOAD-BEARING — do not simplify it away
fᵀM = 0 is not decorative, and it is not merely an internal consistency check. It is the ONLY test in the suite that catches a correspondence whose two views' Jacobians have been SWAPPED — M₂ handed where M₁ belongs, the natural mistake when porting a residual onto bearings.
Nothing else sees it, and that is measured rather than assumed (test/essential_tangent_mc_tests.jl). The swap is very nearly norm-preserving: the two halves of the tangent Sampson's G_x are scaled by |Eᵀf₂| and |Ef₁|, whose ratio has median 1.018 and 5–95% range [0.972, 1.066], so ‖G_x‖ barely moves and the residual stays plausible. Every Monte Carlo calibration cell keeps reporting NEES ≈ 1 — the best of them gets to 0.970, inside its own 4 SE band. Tangency, by contrast, fails loudly: fᵀM measures 4.8e-19 with the right Jacobian against 1.0e-3 with the swapped one, a factor of 2e15.
So a future cleanup that reads the tangency assertion as redundant with the statistical gates would delete the one check that catches a view swap, and the statistical gates would go on succeeding while it did.
Construction, given that a camera's forward map is the differentiable one: with B an orthonormal basis of T_f S² (s2_tangent_basis, shared with EssentialManifold5 rather than re-derived) and A = ∂project(model, ·)/∂f · B the 2×2 restriction of the projection differential to that plane, M = B·A⁻¹. The projection differential is taken by automatic differentiation through the camera's OWN project, so every forward lens — Kannala-Brandt, double sphere, the OpenCV models — is covered without a per-model derivation, the same derivation-risk-free stance retract_jacobian takes. A BACKWARD lens (a LiftLens — division, Scaramuzza — or a SplineLens), whose back-projection is the closed form and whose projection a solve, instead differentiates backproject itself: M = ∂backproject(lens, x)/∂x · ∂x/∂u at x = K⁻¹u, by the same automatic differentiation, or analytically where the lens has its own (SplineLens).
This is PoseLib's unproject_with_jac; GTSAM's nearest relative is PinholeBase::backprojectPointAtInfinity(p, Dresult_dpose, Dresult_dpoint), which returns a Unit3. GTSAM spells the pixel → calibrated step calibrate, which collides with THIS stack's task verb calibrate (estimating a camera), so the name here stays backproject.
Returns (NaN bearing, NaN Jacobian) wherever backproject does — outside the lens's domain, past a fold, or where the tangent restriction A is singular (a ray the camera cannot resolve in two directions). Never throws.
VisualGeometryToolkitCore.Cameras.bearing_correspondences — Method
bearing_correspondences(camera1, camera2, cs_px) -> CorrespondencesPixel correspondences cs_px (u => v, view 1's pixel to view 2's) as bearing correspondences: each endpoint back-projected by its camera (backproject_with_jacobian(camera, u) -> (f, M)), the unit bearings the pair and the 3×2 pixel Jacobians its M₁/M₂ attribute columns, which put a bearing residual back in pixels (bearings.md). One back-projection per endpoint, no filtering: a pixel a camera does not back-project gives its non-finite bearing.
VisualGeometryToolkitCore.Cameras.calibration_matrix — Method
calibration_matrix(model::CameraModel{<:LogicalIntrinsics}) -> SMatrix{3,3,T,9}Return the 3×3 calibration matrix K as a plain SMatrix, preserving the intrinsics' stored element type (Float64 for the default constructors, Float32 for e.g. LogicalIntrinsics(CameraCalibrationMat{Float32}(…))). A projective camera (ProjectiveIntrinsics) has no calibration matrix: there is no method, so the call raises a MethodError. For a SplineLens camera K is the lens's gauge — the pinhole at the pixel imaging ẑ (skew 0) — kept as it is by every fit.
Example
model = CameraModel(; focal=800.0, pp=(320.0, 240.0))
K = calibration_matrix(model) # [800 0 320; 0 800 240; 0 0 1]VisualGeometryToolkitCore.Cameras.distortion_map — Method
distortion_map(model::CameraModel{<:AbstractIntrinsics, <:AbstractLensModel}) -> TransformationMath-space transform mapping undistorted COLMAP pixel coordinates to distorted ones: pixel → K⁻¹ → lens → K → pixel. Spelled as a plain composition (affine_part(K) ∘ lens ∘ affine_part(K⁻¹)) so image_warp's fuse merges the calibration affines into the COLMAP/Julia index bridges — the fused per-pixel path is one affine, the lens polynomial, one affine. For a SplineLens camera K is the gauge pinhole at the optical-axis pixel, so the undistorted frame is that pinhole's.
VisualGeometryToolkitCore.Cameras.distortion_map — Method
distortion_map(m::RadialHomography) -> TransformationThe radial model's undistorted → distorted pixel map, c + μ·(u − c) with the division model's μ (project(m, (u − c, 1)) as a map): the camera bridge's distortion_map, distortion_map(CameraModel(m)). Its inverse is the undistortion, inv(distortion_map(m)); NaN where the model's field ends (as its lens does), never nothing.
VisualGeometryToolkitCore.Cameras.division_bounded — Method
division_bounded(model::CameraModel{<:Union{LogicalIntrinsics, ProjectiveIntrinsics}, <:LiftLens}, sz; margin = HORIZON_MARGIN) -> BoolDoes the whole image lie inside margin · monotone_radius(lens) (in the lens's normalized distorted radius; a projective camera's is centred pixels)? The general-family form of the coefficient-level predicate above.
VisualGeometryToolkitCore.Cameras.division_bounded — Method
division_bounded(k, center, sz; margin = HORIZON_MARGIN) -> BoolWhether the perspective (undistorted) image of a frame of size sz = (width, height) is BOUNDED under a division model k about center (all in the same pixel units — e.g. a RadialHomography's k/center). For k < 0 the undistorted radius r_u = r_d/(1 + k r_d²) diverges at the 90° circle r_∞ = 1/√(−k): content at or beyond it undistorts to infinity, and no finite canvas holds it. True iff the farthest image corner's distorted radius stays inside margin·r_∞ (k ≥ 0 is always bounded).
When true, the corner stretch factor is 1/(1 + k·r_max²) — the canvas scale an undistorted image needs to keep every input pixel in bounds.
VisualGeometryToolkitCore.Cameras.fov_radius — Method
fov_radius(model, θ) -> Float64The UNDISTORTED (perspective-plane) pixel radius of the cone of half-angle θ radians: f·tan(θ). The companion of horizon_radius — one says where the lens stops imaging, this says how big a chosen field of view is once flattened.
A projective camera (ProjectiveIntrinsics) has no focal length, so no field of view: its radius is an ArgumentError (and so is a finite max_fov in undistorted_rect, which reads it).
VisualGeometryToolkitCore.Cameras.horizon_radius — Method
horizon_radius(model) -> Float64The PIXEL radius of the camera's horizon: the circle at which rays reach the lens's monotone limit, beyond which nothing can be placed on a perspective plane at all. Inf for a lens with no such limit (a pinhole, a mild polynomial). This is the circle plot_rectification draws on the raw panel, and the guide for how much of a frame it is meaningful to rectify.
A projective camera (ProjectiveIntrinsics) has no focal length: its radius is read in its own centred-pixel frame, the unit-focal one its lens coefficients are written in.
It is NOT a safe rectification bound by itself. The distorted radius is very nearly linear in the view angle, while the perspective magnification is tan θ, so a generous FRACTION of this radius is still a fraction of a degree from the pole: for a 927 px fisheye with a 1419 px horizon, 92% of it is θ ≈ 89.7°, tan θ ≈ 199, and the rectified plane comes out 369 000 px wide. Bound the ANGLE instead — see undistorted_rect's max_fov.
VisualGeometryToolkitCore.Cameras.normalized_point — Method
normalized_point(intrinsics::AbstractIntrinsics, 𝐩::StaticVector{2}) -> SVector{2}The normalized (distorted) image point x = K⁻¹𝐩 of the pixel 𝐩: the first two coordinates of K⁻¹ (𝐩, 1), K the intrinsics' pixel matrix (skew included). It is the pixel-frame half of backproject — a lens's backproject(lens, x) takes it to the bearing — and the point a spline model's lattice is laid over (spline_window(area, grid, normalized_point(K, 𝐩))). The one spelling of that map: the scalar back-projection, the batched kernels and a caller reading a spline lattice at a pixel all call it. Allocation-free, in the element type of K⁻¹ and 𝐩 (Float32, ForwardDiff Duals), callable in a device kernel.
VisualGeometryToolkitCore.Cameras.pinhole — Method
pinhole(model::CameraModel) -> CameraModel{I, Pinhole}
imaging_transform(model::CameraModel)The lens-stripped pinhole twin (downstream packages hand-rolled CameraModel(x.intrinsics, Pinhole()) in three places) and the composed camera-space → pixel map as a value. intrinsics/lens accessors live in cameras.jl. For a SplineLens camera the twin is the gauge pinhole at the optical-axis pixel (the pixel imaging ẑ).
VisualGeometryToolkitCore.Cameras.radial_homography — Method
radial_homography(camera::CameraModel{<:LogicalIntrinsics, <:Division}, H::HomographyMat) -> RadialHomographyThe radial homography of a plane homography H (plane point → undistorted pixel) seen through a METRIC division camera: RadialHomography(H, λ, c) with λₚ = kₚ / f^{2p}, kₚ the camera's division coefficients in K's normalized chart, f its focal length (pixels) and c its principal point. The projective method's λ's are pixel-frame; this converts the metric camera's to them, so radial_transfer(radial_homography(camera, H), X) is the raw pixel project(camera, K⁻¹·H·[X; 1]) of a plane point X. Throws an ArgumentError for fx ≠ fy or a skewed K, whose lens is not radially symmetric in pixels.
VisualGeometryToolkitCore.Cameras.radial_homography — Method
radial_homography(camera::CameraModel{<:ProjectiveIntrinsics, <:Division}, H::HomographyMat) -> RadialHomographyThe radial homography of a target's plane homography H seen through a projective division camera: RadialHomography(H, camera). A TargetHomographies entry (a HomographyMap) pairs with the camera that measured it through the HomographyMap method below.
VisualGeometryToolkitCore.Cameras.radial_homography — Method
radial_homography(camera::CameraModel{<:Union{ProjectiveIntrinsics, LogicalIntrinsics}, <:Division}, h::HomographyMap{3}) -> RadialHomographyThe radial homography of a plane → pixel HomographyMap (a TargetHomographies entry, labeled or not): radial_homography(camera, H) of its matrix. The map's frames are not checked against the camera, which holds none.
VisualGeometryToolkitCore.Cameras.reframed_calibration — Method
reframed_calibration(S::SimilarityMap{2}, K::CameraCalibrationMat) -> CameraCalibrationMat{Float64}
reframed_calibration(from::Type{<:PixelFrame}, to::Type{<:PixelFrame}, K::CameraCalibrationMat) -> CameraCalibrationMat{Float64}THE change of image frame of a calibration matrix: K in S's source frame becomes to_homogeneous(S)·K in its destination frame, so a pixel K·x̃ of the source is S(K·x̃) of the destination. In Float64. A similarity of the image plane keeps K's form when its rotation is the identity (a scale s multiplies the focals, the skew and the principal point, then the translation moves the principal point); a rotated S gives no upper-triangular K and is refused by K's own structure check (ArgumentError). inv(S) brings K back.
The 3-argument form is the pixel-frame change, S = SimilarityMap(EuclideanMap(from, to)) (a translation by the frames' half-pixel offset; the focal lengths and the skew kept): CameraModel(K; pixel_frame) and VisualGeometryToolkitIO's stored K (its readers and writers) all move a K through it. VisualGeometryToolkitFeatures' half-resolution RAW images move K through the map form with their frame map (raw_frame_map). K was checked (or deliberately not, check = false) when it was built; a positive scale keeps its focals positive, so only the structure is re-checked.
K = CameraCalibrationMat(800.0px, [320.0px, 240.0px]) # OpenCV's frame
reframed_calibration(PixelFrame{:opencv}, PixelFrame{:colmap}, K) # cx, cy = 320.5, 240.5VisualGeometryToolkitCore.Cameras.reprojection_error — Method
reprojection_error(camera, pose, X, u) -> SVector{2}The reprojection residual of a scene point, Jacobian-free: project(camera, pose(X)) − u, with pose any map into the camera frame (a world → camera EuclideanMap, X_c = R·X + t; a camera → world pose enters as inv(pose), which is x ↦ Rᵀ(x − t)). NaN where the lens cannot image the point (project answers on the ray domain). Element type follows the inputs, so ForwardDiff duals flow through the pose and the lens.
VisualGeometryToolkitCore.Cameras.undistort! — Method
undistort!(dst, [T = Float64], model, src, rect; fillvalue = zero(eltype(dst))) -> dstGENERAL form: resample the undistorted-plane window rect (COLMAP pixel coordinates) into dst. dst's size sets the sampling density — supply a smaller dst to downscale; the scale and offset fold into the fused warp affines. The rect-free methods above render Rect(0, 0, w, h) at unit scale (the classic same-size undistortion, which CROPS any content the undistortion pushes outside the frame); pair this form with undistorted_rect to render the full extent instead.
VisualGeometryToolkitCore.Cameras.undistort! — Method
undistort!(dst, [T = Float64], model, src; fillvalue = zero(eltype(dst))) -> dstIn-place undistort: resample src into the pre-allocated dst through the fused lens warp, on the backend that owns dst (any KernelAbstractions backend — supply CuArrays for GPU execution). dst and src may differ in eltype (see image_warp!). T sets the warp math's float type: undistort!(dst, Float32, model, src) keeps the per-pixel math single-precision for GPU throughput. The order is Julia Base's: the mutated dst, then the type, then the inputs.
VisualGeometryToolkitCore.Cameras.undistort — Method
undistort(model, img, rect; scale = 1, fillvalue = zero(eltype(img)), device = default_device()) -> FramedImageAllocating rect form of undistort!: the canvas is sized ceil.(widths(rect) ./ scale), rect is resampled into it, and the result is returned WITH its placement as a FramedImage — rect is the window of the camera's pixel frame the canvas covers, so imshow(pos, undistort(model, img, ext; scale)) is self-placing and the ext/scale pair never travels beside the pixels by hand. The in-place undistort! forms are unchanged (the caller owns dst and its placement); the rect-free same-size forms return bare matrices (their placement is trivially the frame). device as in undistort: the warp runs there, the canvas comes back in img's array kind.
VisualGeometryToolkitCore.Cameras.undistort — Method
undistort(model::CameraModel{<:AbstractIntrinsics, <:AbstractLensModel}, img)Remove lens distortion from an image.
For each output (undistorted) pixel, computes the corresponding source (distorted) pixel through model.lens and resamples. Uses VGC's image_warp builder to bake pixel coordinate conventions into a BoundedMap (K is assumed to be in COLMAP convention: image corner at origin, pixel centers at 0.5).
Each output pixel is sampled through project, which NaNs a ray past a radial factor's fold (the pixel takes the fill value) but does not test the other folds: past a fold of an Additive's deltas or a composed distortion's Tilt the pixel samples a finite source pixel whose backproject is a different ray, so the output repeats content of the imaged branch there. Where the lens overlaps itself over the sensor, no undistortion is one-to-one, and a map built from backproject may jump between preimages (see backproject): injectivity is checked on the lens over the sensor rectangle, not per pixel. A SplineLens camera undistorts onto its gauge pinhole (its K: the pinhole at the pixel imaging ẑ), in every form here and in undistort!.
Arguments
model::CameraModel: Camera model (any lens family)img: Input distorted image (any AbstractMatrix — Gray, RGB, etc.)device: the KernelAbstractions backend the warp runs on, by defaultdefault_device()(an accelerator when one is functional, elseKA.CPU()).imgis uploaded to it once, and the result comes back inimg's array kind: anArraygives anArray, a device array stays on its device with no copy. The in-placeundistort!runs ondst's backend.
Returns
Undistorted image of the same size and element type.
Example
model = CameraModel(; focal=2458.0, pp=(2099.0, 1494.0),
lens=Lens(Perspective(), Additive(Radial((-0.159, 0.048)), Tangential(0.006, -0.001))))
img_clean = undistort(model, img)VisualGeometryToolkitCore.Cameras.undistorted_rect — Method
undistorted_rect(m::RadialHomography, (width, height); margin = HORIZON_MARGIN, max_fov = Inf) -> RectThe undistorted-plane extent of a width×height frame under m's division model (the walk of its CameraModel(m)): the frame boundary undistorted (u = c + (d − c)/(1 + Σₚ λₚ‖d − c‖²ᵖ)), with boundary radii at or beyond margin·r_∞ (the lens's horizon; r_∞ = 1/√(−k) for one coefficient) clipped to it — the honest cap when division_bounded is false and the true extent is infinite. Feed the rect to the rect form of undistort/undistort!:
ext = undistorted_rect(m, (w, h))
canvas = undistort(CameraModel(m), img, ext;
scale = max(1, GeometryBasics.widths(ext)[1] / 5000))max_fov (radians) additionally bounds the extent by FIELD OF VIEW, capping it at the bounding square of that cone about the principal point. Inf, the default, leaves the frame's own field alone; a finite value is what makes a fisheye — whose horizon can sit inside the frame — rectify to a usable plane at all, since tan θ is unbounded in the last degree before the pole. DEFAULT_MAX_FOV is the recommended cap. A field of view needs a metric camera: for a radial homography (a projective camera, ProjectiveIntrinsics) a finite max_fov is an ArgumentError.
VisualGeometryToolkitCore.Cameras.valid_rect — Method
valid_rect(m::RadialHomography, (width, height); margin = HORIZON_MARGIN) -> Rect
valid_rect(model::CameraModel{<:Union{LogicalIntrinsics, ProjectiveIntrinsics}, <:AbstractLensModel}, sz;
margin = HORIZON_MARGIN) -> RectThe INSCRIBED counterpart of undistorted_rect — OpenCV's alpha = 0 endpoint: the largest axis-aligned window of the undistorted plane containing NO invalid pixels, so undistort(model, img, valid_rect(model, size)) renders with no fill — the no-black-borders mode. Same frame-boundary walk as undistorted_rect (one authority, _boundary_undistort), but taking per-edge extrema INWARD — the left constraint is the maximum x over the undistorted left edge, and so on — instead of the global bounding box (sampled, like the outer rect; corners are the edge-sample endpoints, so both bow directions are covered).
Boundary radii at or beyond margin·r_∞ clip to the horizon exactly as in undistorted_rect: when the whole horizon circle lies inside the frame the valid region is unbounded and the clip is the honest cap. Throws when the inscribed window is empty (boundary folded past the principal point). A SplineLens camera's rect is on its gauge pinhole (the pixel imaging ẑ).
VisualGeometryToolkitCore.backproject — Method
backproject(intrinsics::AbstractIntrinsics, lens::AbstractLensModel, u::StaticVector{2,T<:Real}) -> Vec3The pixel-frame half of back-projection: K⁻¹ takes the pixel u to the normalized distorted point, and the bearing is the lens's own backproject(lens, xd) — which, not this method, is what a lens family extends.
VisualGeometryToolkitCore.backproject — Method
backproject(model::CameraModel, pixels::AbstractVector{<:StaticVector{2}}; device = get_backend(pixels)) -> rays
backproject_with_jacobian(model::CameraModel, pixels::AbstractVector{<:StaticVector{2}}; device = get_backend(pixels))
-> Vector{Tuple{SVector{3}, SMatrix{3, 2}}}The batched backproject (and backproject_with_jacobian) on the KernelAbstractions backend device (by default the pixels' own, get_backend(pixels): a host vector stays on the host, a device array on its device; give default_device() to send host pixels to the accelerator): the scalar map per pixel, one pointwise! over a device copy of the model and the pixels, each ray an SVector{3}, the result back in the pixels' array kind — bit for bit the scalar value on the CPU. One kernel on every backend: the model goes in its device form (isbits, or its adapt rule); a model with neither fails on an accelerator. A SplineLens camera's batched backproject is its own kernel (spline_lens.jl).
VisualGeometryToolkitCore.backproject — Method
backproject(model::CameraModel, u::StaticVector{2,T<:Real}) -> Vec3Backproject 2D pixel coordinates to 3D ray direction.
Inputs and outputs are unitless (assumed units: px for input, unitless direction for output). The output eltype follows the usual promotion of the input and the intrinsics: Float64 pipelines are unchanged; a Float32 camera (LogicalIntrinsics{Float32}) with Float32 pixels stays Float32; ForwardDiff Dual pixels give a Dual direction, so the bearing can be differentiated with respect to the pixel; integer pixels give the bearing of their floating-point value, as the lens-level backproject(lens, xd) and the grid models' backproject do.
The return type and semantics depend on the intrinsics type:
- LogicalIntrinsics: Returns normalized unitless direction
Arguments
model::CameraModel: Camera model with intrinsics and lensu::StaticVector{2,T}: 2D pixel coordinates [u, v] (unitless, in px)
Returns
Vec3: Normalized ray direction (unitless)
Example
camera = CameraModel(16.0mm, Size2(width=5.86μm/px, height=5.86μm/px), [320.0px, 240.0px])
u = SVector(400.0, 300.0) # px (unitless)
ray = backproject(camera, u) # Returns normalized Vec3 directionVisualGeometryToolkitCore.project — Method
project(model::CameraModel, X::AbstractVector) -> Point2Project 3D point X in camera coordinates to 2D pixel coordinates.
Arguments
model::CameraModel: Camera modelX::AbstractVector: 3D point in camera coordinates [X, Y, Z] (with units)
Returns
- Point2 with pixel coordinates (including px units)
Example
model = CameraModel(16.0mm, Size2(width=5.86μm/px, height=5.86μm/px), [320.0px, 240.0px])
X = SVector(10.0mm, 5.0mm, 100.0mm)
u = project(model, X) # Returns Point2 in pixelsVisualGeometryToolkitCore.unproject — Method
unproject(model::CameraModel{<:LogicalIntrinsics}, u::AbstractVector, depth::Real) -> Point3
unproject(intrinsics::LogicalIntrinsics, lens, u, depth) -> Point3The camera-frame point on the ray behind pixel u whose Z-DEPTH is depth — the convention of depth images: distance along the optical axis. It is b·(depth / b₃) with b = backproject(intrinsics, lens, u), for every lens, and NaN where the ray has b₃ ≤ 0: a Z-depth names no point on such a ray. A point at distance d ALONG any ray, past 90° included, is d · backproject(model, u). A projective camera has no metric ray, so no Z-depth: there is no method, and the call raises a MethodError.
VisualGeometryToolkitCore.Cameras.angle_of_view — Method
angle_of_view(camera::BoundedCamera; dimension = :horizontal) -> θThe angle of view of a bounded camera: the FULL angle, in radians, of the scene it IMAGES in dimension — its pixel extent, bounded by its lens_coverage (where the lens covers the sensor) — read through the camera's own backproject (K⁻¹, then the lens), so a distorted or fisheye lens answers its true angle and an anisotropic K uses the focal length of that axis. For a pinhole with a centred principal point it is the inverse of focal_length(resolution, θ; dimension), which takes the same full angle (focal_length(sensor, θ) is its metric form). (The lens's own limit, view_angle_limit, is a HALF angle from the optical axis.) The size this angle spans at a distance is field_of_view.
The sample points, each moved in along the line from the principal point to where the lens coverage ends when it lies past it (a circular fisheye's dark corners, the 14 mm disc cut by a wide sensor):
:horizontal: the two edge points on the principal point's row,(0, c_y)and(width, c_y);:vertical: the two on its column,(c_x, 0)and(c_x, height);:diagonal: the four corners of the extent.
The answer is twice the SMALLEST of those points' view angles — the nearer edge when the principal point is off-centre, so the cone of that full angle about the optical axis is imaged in that dimension. A camera whose whole sensor is covered (the default coverage) reads the sensor's edges. A principal point outside the coverage images nothing about the axis: an ArgumentError. For a lens that is not radially symmetric the valid_domain is the star polygon through each azimuth's fold, pulled in by at most its chords' sag (R·10⁻⁵), so the answer can understate the angle by that much.
A SplineLens camera reads about its gauge principal point, the pixel imaging ẑ, so the angles are about the optical axis.
Clipping by the model: a point outside the lens model's valid_domain (or one the lens does not image) counts as the lens's own limit, so when the valid domain clips the covered area in that dimension the answer is 2 · view_angle_limit(lens).
cam = BoundedCamera(CameraModel(K; lens = Lens(Equidistant())), sensor("Sony", "IMX250"))
angle_of_view(cam) # horizontal, full angle in radians
angle_of_view(cam; dimension = :diagonal)VisualGeometryToolkitCore.Cameras.field_of_view — Method
field_of_view(camera::BoundedCamera, distance; dimension = :horizontal) -> sizeThe field of view of a bounded camera at distance: the SIZE of the scene it images there, on a plane perpendicular to its optical axis — 2·distance·tan(θ/2) for its angle_of_viewθ in dimension, in the unit of distance (a length such as 2.0m, or a plain number). An angle of view of 180° or more spans no finite size on a plane, and a distance must be positive: both an ArgumentError. (Not the field-of-view LENS model FieldOfView, Devernay's distortion — a different thing that shares the words.)
field_of_view(cam, 3.0m) # the width of the scene imaged 3 m away
field_of_view(cam, 3.0m; dimension = :vertical)VisualGeometryToolkitCore.Cameras.lookat — Method
lookat(eye::StaticVector{3, <:Real}, target::StaticVector{3, <:Real}, up::StaticVector{3, <:Real};
convention::Symbol = :cv)Create a camera extrinsics transform that looks from eye position toward target.
Unitless version for internal use and backwards compatibility.
convention selects the camera-frame convention (same symbol-selector precedent as Rect(sensor, :julia)):
:cv(default) — Computer Vision convention:- +Z axis points FORWARD (toward target)
- +X axis points RIGHT
- +Y axis points DOWN
- Camera -Y aligns with world up direction (supply
upas [0, -1, 0])
:gfx— graphics convention (Mitsuba/OpenGL-stylelook_atframe): camera-to-world columnsz = normalize(target − eye),x = normalize(cross(up, z)),y = cross(z, x); supplyupas the world up direction (typically [0, 1, 0]). The returned map is still the world-to-camera extrinsics — only the frame construction differs.
Arguments
eye::StaticVector{3}: Camera position in world coordinates (plain numbers, in the caller's unit)target::StaticVector{3}: Point to look at in world coordinates (same unit)up::StaticVector{3}: World up direction (see per-convention notes above)
Returns
CameraExtrinsics: World-to-camera transformation (frame-labeledEuclideanMap)
Example
# Camera at (0, 0, 0) looking toward (0, 0, 1000), with world -Y pointing up
extrinsics = lookat(SVector(0.0, 0.0, 0.0), SVector(0.0, 0.0, 1000.0), SVector(0.0, -1.0, 0.0))See also lookat_pose for the camera-to-world form (bit-exact eye translation, no inversion round-trip).
VisualGeometryToolkitCore.Cameras.lookat — Method
lookat(eye::StaticVector{3, <:Quantity}, target::StaticVector{3, <:Quantity}, up::StaticVector{3, <:Real};
convention::Symbol = :cv)Create a camera extrinsics transform that looks from eye position toward target.
A PHYSICAL pose: eye and target are lengths, and the extrinsics' translation is a length in metres — the plain lookat of the two points in metres, read back in metres (a plain point beside a length is a DimensionError).
Uses Computer Vision convention:
- +Z axis points FORWARD (toward target)
- +X axis points RIGHT
- +Y axis points DOWN
- Camera -Y aligns with world up direction
Arguments
eye::StaticVector{3, <:Quantity}: Camera position in world coordinates (physical lengths)target::StaticVector{3, <:Quantity}: Point to look at in world coordinates (physical lengths)up::StaticVector{3, <:Real}: World up direction, unitless (typically [0, -1, 0] for CV convention where -Y is up)
Returns
CameraExtrinsics: World-to-camera transformation (frame-labeledEuclideanMap, physical)
Example
# Camera at (0, 0, 0) looking toward (0, 0, 1000mm), with world -Y pointing up
extrinsics = lookat(SVector(0.0mm, 0.0mm, 0.0mm), SVector(0.0mm, 0.0mm, 1000.0mm), SVector(0.0, -1.0, 0.0))VisualGeometryToolkitCore.Cameras.lookat_pose — Method
lookat_pose(eye::StaticVector{3, <:Real}, target::StaticVector{3, <:Real}, up::StaticVector{3, <:Real};
convention::Symbol = :cv) -> CameraPoseCamera POSE (camera-to-world transform) that looks from eye toward target, as a typed CameraPose — the pose-side companion of lookat, sharing its convention selector (:cv / :gfx).
Built directly from the frame columns (rotation = camera basis as camera-to-world columns, translation = eyebit-exactly) rather than as inv(lookat(...)), so no inversion round-trip touches the translation. Feed the result straight to the frame-typed camera constructor PosedCamera(model, lookat_pose(...)), which inverts internally into the extrinsics slot.
VisualGeometryToolkitCore.focal_length — Method
focal_length(sensor::Sensor, θ::Real; dimension=:horizontal)The physical focal length from the FULL field of view angle and the sensor: focal_length(sensor.resolution, θ) pixels times the pitch of that axis.
Arguments
sensor::Sensor: the sensor (itsresolutionandpitch)θ::Real: the full angle of view in radiansdimension::Symbol: Which dimension the angle of view applies to (:horizontalor:vertical)
Returns
- Physical focal length (a length quantity)
Examples
s = sensor("Sony", "IMX174")
f_mm = focal_length(s, deg2rad(40.0))VisualGeometryToolkitCore.focal_length — Method
focal_length(resolution::Size2, θ::Real; dimension=:horizontal)The focal length an angle of view implies: in pixels from the FULL angle of view (plain radians — angles carry no dimension in the VGC unit system) and resolution. Its inverse, for a camera, is angle_of_view. The other forms: focal_length(sensor::Sensor, θ) (a length) and the accessor focal_length(intrinsics) ((fx, fy) in pixels).
Arguments
resolution::Size2: Image resolution (width, height) in pixels: plain numbers or dimensionless quantities (px); a length is aDimensionErrorθ::Real: the full angle of view in radiansdimension::Symbol: Which dimension the angle of view applies to (:horizontalor:vertical)
Returns
- Focal length in pixels (same logical unit as resolution)
Examples
focal_length(Size2(width=1280px, height=960px), 0.698)
focal_length(Size2(width=1280px, height=960px), deg2rad(40.0))VisualGeometryToolkitCore.p3p — Method
p3p(model::CameraModel, correspondences) -> FixedModels{4, Extrinsics3}Solve the Perspective-3-Point (P3P) problem using correspondences.
This is a thin wrapper around the core p3p(rays, X) solver that accepts correspondences directly. It extracts the 3D world points and 2D image points from the correspondences, backprojects the image points to rays, and calls the core solver.
Arguments
model::CameraModel: Camera model for backprojecting image points to rayscorrespondences: Correspondence container with exactly 3 point pairs (3D world → 2D image); any container satisfying thesources/targetsprotocol (Correspondences, StructArray of Pairs, plain Pair vector)
Returns
FixedModels{4, Extrinsics3}: Stack-allocated container of 0–4 rigid poses. Empty (length == 0) if no valid solution exists. Iterate withfor ext in result ... endorcollect(result).
Examples
# Setup camera
s = sensor("Sony", "IMX174")
sensor_bounds = Rect(s)
f = focal_length(s, deg2rad(40.0); dimension=:horizontal)
pp = centroid(sensor_bounds) .* px
model = CameraModel(f, s.pitch, pp)
# Create correspondences (3D world points → 2D image points)
X3 = [Point3(100.0, 100.0, 0.0), Point3(200.0, 150.0, 0.0), Point3(150.0, 200.0, 0.0)]
u3 = [Point2(640.0, 480.0), Point2(700.0, 500.0), Point2(680.0, 550.0)]
cs = correspondences(X3, u3)
# Solve P3P — always a FixedModels; empty (length 0) when no valid pose
result = p3p(model, cs)
cameras = PosedCamera.(Ref(model), collect(result))See Also
p3p(rays, X): Core P3P solver that works with rays directlysample_p3p: Sample valid camera poses by randomly sampling correspondences
VisualGeometryToolkitCore.Cameras.sample_p3p — Method
sample_p3p(model, X3::SVector{3}, sensor_bounds, plane::Plane3D; max_retries=1000)Sample a P3P configuration using 3 fixed world points with winding-consistent image sampling.
Given 3 world points (e.g., board corners), samples 3 random image points and sorts them with consistent winding (world CCW, image CW due to y-axis flip). This ensures:
- Deterministic scale: World point distances define the scale
- Consistent orientation: Matching winding prevents reflected/flipped solutions
Points can be supplied in any order - the function sorts world points to CCW and image points to CW (matching winding due to y-axis inversion between coordinate systems).
Arguments
model::CameraModel: Camera modelX3::SVector{3,Point3}: 3 world points (any order, will be sorted to CCW)sensor_bounds::Rect: Sensor bounds for sampling image pointsplane::Plane3D: Reference plane for chirality test
Returns
(camera, correspondences): Camera and the 3 correspondences used (in CCW order)
Example
# Board corners in world coords (centered, y-up, z=0)
board_rect = Rect(bundle, :world)
corners = coordinates(board_rect) # 4 corners
X3 = SVector{3}(Point3.(vcat.(corners[[1,2,3]], 0.0))) # any 3 corners
camera, correspondences = sample_p3p(model, X3, sensor_bounds, Plane3D())VisualGeometryToolkitCore.Cameras.sample_p3p — Method
sample_p3p(model, X, sensor_bounds, plane::Plane3D; max_retries=1000)
sample_p3p(rng, model, X, sensor_bounds, plane::Plane3D; max_retries=1000)Sample a random P3P configuration with chirality filtering.
Like sample_p3p(model, X, sensor_bounds) but additionally filters solutions to only those where the camera is on the positive side of the given plane (i.e., signed_distance(plane, camera_center) > 0).
This is useful for planar calibration targets where you want to ensure the camera is "in front of" the target plane.
Arguments
rng=Random.default_rng(): Random number generator (optional first positional argument)model::CameraModel: Camera modelX::Vector{<:StaticVector{3,Float64}}: 3D world points on or near the planesensor_bounds::Rect: Sensor bounds for sampling image points (unitless, in pixels)plane::Plane3D: Reference plane for chirality test
Keyword Arguments
max_retries=1000: Maximum number of sampling attempts
Returns
(camera, correspondences): Tuple containing:camera::PosedCamera: The camera with valid pose (in front of plane)correspondences::StructArray{Pair{Point3, Point2}, 1, ...}: The 3 correspondences used
Example
# World coordinate system: origin at board center, z=0 plane, z-axis pointing out
board_plane = Plane3D() # z=0, normal = (0,0,1)
# Sample camera that is in front of the board (z > 0)
camera, correspondences = sample_p3p(model, X, sensor_bounds, board_plane)VisualGeometryToolkitCore.Cameras.sample_p3p — Method
sample_p3p(model, X, sensor_bounds; max_retries=1000)
sample_p3p(rng, model, X, sensor_bounds; max_retries=1000)Sample a random P3P configuration by sampling from world points X and random image points.
Creates 3D-2D correspondences by:
- Sampling 3 random points from X (world coordinates)
- Sampling 3 random 2D image points in sensor bounds
- Pairing them as correspondences and solving P3P
Takes a logical camera (CameraModel{<:LogicalIntrinsics}): X and the sensor bounds are unitless (the construction boundary stripped units).
Arguments
rng=Random.default_rng(): Random number generator (optional first positional argument)model::CameraModel: Camera modelX::Vector{<:StaticVector{3,Float64}}: 3D world points, unitlesssensor_bounds::Rect: Sensor bounds for sampling image points (unitless, in pixels)
Keyword Arguments
max_retries=1000: Maximum number of sampling attempts
Returns
(extrinsics, correspondences): Tuple containing:extrinsics::Vector{Extrinsics3}: Camera extrinsics (world-to-camera transforms) for valid P3P solutionscorrespondences::StructArray{Pair{Point3, Point2}, 1, ...}: The 3 correspondences used for P3P, backed bySVector{3}columns
Throws an error if no valid configuration is found after max_retries attempts.
Examples
# Setup camera
s = sensor("Sony", "IMX174")
sensor_bounds = Rect(s)
f = focal_length(s, deg2rad(40.0); dimension=:horizontal)
pp = centroid(sensor_bounds) .* px
model = CameraModel(f, s.pitch, pp)
# 3D points (planar points at z=0)
X = [Point3(100.0, 100.0, 0.0), Point3(200.0, 150.0, 0.0), ...]
# With default RNG
extrinsics, correspondences = sample_p3p(model, X, sensor_bounds)
# With explicit RNG for reproducibility
rng = Random.MersenneTwister(12345)
extrinsics, correspondences = sample_p3p(rng, model, X, sensor_bounds)
# Access correspondence data (StructArray interface)
X_used = correspondences.first # All 3D points
u_used = correspondences.second # All 2D image points
# Or index individual correspondences: correspondences[1].first, correspondences[1].second
# Use best extrinsics (e.g., based on reprojection error)
cameras = PosedCamera.(Ref(model), extrinsics)
best_camera = cameras[1] # Or select based on reprojection errorIntrinsics and Sensors
The intrinsics and sensor types are on the Intrinsics and Sensors API page.
Boards and canvases
A planar target's Board is the sensor's analogue: the printed board's size (lengths, or plain numbers for a unitless target) and its corner in the design chart; Rect(board) is its domain there, as Rect(sensor) is the image's in pixels. The Canvas is the printed page the board lies on, with the image printed on it (CanvasTexture: a relative path, its SHA-256 and the rectangle it covers) — canvas ⊃ board ⊃ pattern, all rectangles of the one design chart the Canvas docstring states.
VisualGeometryToolkitCore.Cameras.Board — Type
Board{T}The calibration board a planar target's pattern is printed on, the target's analogue of a Sensor: its size and where it lies in the design chart. It follows the target's unit rule: physical (lengths, stored in metres) for a physical target, plain numbers for a unitless one.
Fields
size::Size2{T}: the board proper, width × height — BlobBoards'board_size, the printed board inside the sheet's white margin, frame and corners included; NOT itscanvas_size(the sheet) and not the pattern's extent. Lengths (a quantity of any other dimension is aDimensionError) or plain numbers; a mix is anArgumentError.origin::SVector{2, Float64}: the board's lower-left corner in the design chart (the coordinates of the nominal points, stated once atCanvas): plain metres for a physical board, the design's plain numbers for a unitless one.
Read them as properties, board.size and board.origin, as a sensor's resolution and pitch are.
Constructors
Board(Size2(width = 297mm, height = 210mm)) # corner at the design origin
Board(Size2(width = 297mm, height = 210mm), [-10.0mm, -10.0mm]) # the corner as lengths
Board(Size2(width = 297mm, height = 210mm), SVector(-0.01, -0.01)) # or plain metres
Board(Size2(width = 297.0, height = 210.0), [-10.0, -10.0]) # a unitless boardRect(board) is the board's domain in the design chart in the board's own numbers (plain metres when physical) — what a target's surface is bounded by (grown by a 16-eps rounding margin), as Rect(sensor) is the image's domain in pixels. ustrip(u, board) and with_unit(board, u) move it between the two forms, as they move a target's points.
See Also
aspect_ratioPlanarTarget,boardCanvas: the printed page the board lies on
VisualGeometryToolkitCore.Cameras.Canvas — Type
Canvas{T}The page a planar target is printed on: the sheet that holds the Board (the calibrated rectangle) and around it the margin; the pattern is what is printed on the board. A canvas ⊃ board ⊃ pattern hierarchy, the canvas being the target's record of what was printed, not of what was calibrated.
The design chart
The board, the canvas and a canvas texture are axis-aligned rectangles of ONE chart, the design chart: the plane coordinates of a planar target's nominal points, x to the right and y up as the printed side is seen from the front, in metres for a physical target (lengths at the constructors, plain metres stored) and in the design's plain numbers for a unitless one. Each rectangle is its lower-left corner (origin) and its size; the board's placement on the canvas is board.origin − canvas.origin.
Fields
size::Size2{T}: the page, width × height, by the board's length rule (lengths stored in metres, a quantity of any other dimension aDimensionError; or plain numbers; non-negative).origin::SVector{2, Float64}: its lower-left corner in the design chart.texture::Union{Nothing, CanvasTexture{T}}: the printed image (CanvasTexture), ornothing; in the canvas's form (FlexUnits'DimensionErrorotherwise).
Constructors
Canvas(Size2(width = 297mm, height = 210mm), [-20.0mm, -15.0mm])
Canvas(Size2(width = 297mm, height = 210mm), [-20.0mm, -15.0mm];
texture = CanvasTexture("canvas.png", sha, Size2(width = 297mm, height = 210mm), [-20.0mm, -15.0mm]))
Canvas(Size2(width = 297.0, height = 210.0)) # a unitless canvasRect(canvas) is its rectangle in its own numbers. ustrip(u, canvas) and with_unit(canvas, u) move it (and its texture) between the two forms, as they move a board. A PlanarTarget carries it beside its board (canvas = …; canvas(t) reads it).
VisualGeometryToolkitCore.Cameras.CanvasTexture — Type
CanvasTexture{T}The image printed on a Canvas: a reference to the image file and the rectangle of the design chart it covers. VGC never reads the image; the reference is what a reader resolves and checks.
Fields
path::String: the image file, RELATIVE to the file that names it (a rig or board file); an absolute path is anArgumentError.sha256::String: the SHA-256 of the image file's bytes, 64 lowercase hex digits (anArgumentErrorotherwise) — the reader's integrity check.size::Size2{T},origin::SVector{2, Float64}: the rectangle the image covers, by the board's rule (lengths stored in metres or plain numbers; its lower-left corner in the design chart, plain metres when physical). The size is positive.
The image's first row lies along the rectangle's top edge (largest y) and its first column along the left edge (smallest x); its outer pixel edges lie on the rectangle's edges, so each pixel of a W × H image covers width/W × height/H of the chart.
Constructors
CanvasTexture("board.png", sha, Size2(width = 297mm, height = 210mm)) # corner at the design origin
CanvasTexture("board.png", sha, Size2(width = 297mm, height = 210mm), [-10.0mm, -5.0mm])
CanvasTexture("board.png", sha, Size2(width = 297.0, height = 210.0), [-10.0, -5.0]) # a unitless oneRect(texture) is the covered rectangle in its own numbers (plain metres when physical).
The overlap of a rotating camera's views
view_overlap measures, for every pair of frames of a camera turning about its centre, the solid angle of the directions both image and its share of each frame's footprint: the overlap a panorama's pair selection reads.
VisualGeometryToolkitCore.Cameras.view_overlap — Method
view_overlap(cameras; boundary_points = 512) -> StructVector
view_overlap(a, b; boundary_points = 512) -> NamedTupleThe directions two frames of a camera turning about its centre both image, for every pair i < j of cameras: one row (; i, j, solid_angle, fraction_i, fraction_j) per pair, K(K − 1)/2 rows in the order (1, 2), (1, 3), …, (2, 3), … as a StructVector, so rows.solid_angle is a column. solid_angle is the solid angle (sr) of the intersection of the two frames' footprints on the sphere; fraction_i is it over frame i's own footprint, and fraction_j over frame j's. A pair of cameras a, b gives the one row, i = 1, j = 2. For a panorama's pair selection: a pair worth matching shares a fraction of its view.
Each camera is a BoundedCamera of an OrientedCamera, a frame posed by its orientation alone, BoundedCamera(PosedCamera(model, relabel(CameraFrame, WorldFrame, RotationMap(R_k))), extent) for a camera → world rotation R_k, d_world = R_k · d_camera, whose model is a CameraModel (another model, a grid one, is an ArgumentError). Frames may have different cameras. A camera posed by an EuclideanMap has a position, and an unposed one no orientation: both are an ArgumentError naming the fix. Frame k's footprint is the set of directions R_k · backproject(model, u) over its region Ωk, [`imagedregion](@ref)(camera): the sensor ∩ the camera's lens coverage (its image circle, [lenscoverage](@ref)) ∩ where the lens model is valid ([validdomain`](@ref)). Each frame carries its own lens coverage; a camera whose whole sensor is lit gives the extent where the lens model holds, which is not where light falls for a lens whose image circle lies inside the sensor.
The pair (i, j) is computed once, in frame i's image:
- Ωi is a ring of `boundarypoints` pixels along its boundary: the extent's edges (every corner exact), the circle's samples, and their convex intersection by Core's Sutherland–Hodgman.
- Pj is frame
j's ring drawn into imagei: each pixel back-projected byj, rotated by `Riᵀ Rjand projected byi, each edge split into geodesic pieces no longer in imagei` than Ωi's own edges. A direction frameidoes not image (behind a perspective camera, past a lens's fold, NaN, or one whose pixel does not back-project to it within 10⁻⁶ rad) has no pixel; the ring is cut there and continued along a far circle abouti's principal point, four times Ωi's radius, at the direction's azimuth, so it stays a closed curve that runs outside Ωi wherevericannot draw it. - Sutherland–Hodgman clips Pj by the convex Ωi. A curve of any shape keeps its winding number inside the clip, so the clipped ring's area is the integral of Pj's winding number over Ωi; pieces a non-convex Pj splits into come back joined by edges along Ωi's boundary, traversed once each way, which add nothing.
- That area is a solid angle: the clipped ring's vertices are back-projected by
ito unit bearings and joined by geodesics, and their fan of signed spherical triangles fromi's optical axis is summed (Van Oosterom & Strackee,tan(E/2) = a·(b × c) / (1 + a·b + b·c + c·a)). Along Ω_i's boundary the polygon follows the footprint's own geodesic edges: a vertex there is moved onto its edge's great circle, and a clipped edge along a sensor side gets the ring's vertices between its ends (a lens bends the side). - The detours of item 2 are where the winding can be wrong, and only by an integer
nconstant over Ωi: a detour outside Ωi can only wind around all of it. One interior point of Ωi, the candidate farthest from Pj, fixesn: its winding number against whether framejimages its direction (project byj, in Ωj, back-projected within 10⁻⁶ rad). The overlap is the area lessntimes framei's footprint. The same correction covers a footprintjthat wraps framei's backward direction (a wide fisheye behind a narrow lens), whose ring winds the other way round Ωi.
Each footprint's own solid angle is the same fan over its ring from its own optical axis, which needs no hemisphere: the sum is the enclosed area for any apex whose antipode is outside the region, and a footprint never reaches its camera's backward axis. So a lens past 180° (an 8 mm fisheye) is handled; a footprint whose ring comes within 0.05 rad (≈ 3°) of its camera's backward axis is an ArgumentError, as is a region whose boundary the lens cannot back-project and a principal point outside its region.
A perspective camera maps geodesics to lines, so its footprints are exact great-circle polygons and their overlap is exact at any boundary_points. A curved boundary (an image circle, a distorted side) is a polygon of boundary_points vertices, so the error is the chords' sag, O(1/boundary_points²): 1.6 × 10⁻⁵ sr of a 0.77 sr cap at 512 (2 × 10⁻⁵ relative), a quarter of it at 1024. A footprint that coincides with another (the same camera and rotation) clips to itself: fraction 1. A pair whose footprints' bounding caps about the optical axes are disjoint is 0 without clipping.
cam = CameraModel(K; lens = KannalaBrandt(-0.135))
bare(R) = BoundedCamera(PosedCamera(cam, relabel(CameraFrame, WorldFrame, RotationMap(R))), sensor("Sony", "IMX250"))
frame(R) = (b = bare(R); with_lens_coverage(b, image_circle(b, 6.0mm))) # its lens coverage: the datasheet's 6 mm circle
frames = [frame(RotY(2π * k / 11)) for k in 0:10] # 11 frames about the vertical axis
rows = view_overlap(frames)
rows[1].fraction_i # the share of frame 1's view frame 2 seesIndex
VisualGeometryToolkitCore.CamerasVisualGeometryToolkitCore.Cameras.DEFAULT_MAX_FOVVisualGeometryToolkitCore.Cameras.BoardVisualGeometryToolkitCore.Cameras.BoundedCameraVisualGeometryToolkitCore.Cameras.BoundedCameraVisualGeometryToolkitCore.Cameras.CameraModelVisualGeometryToolkitCore.Cameras.CameraModelVisualGeometryToolkitCore.Cameras.CameraModelVisualGeometryToolkitCore.Cameras.CameraModelVisualGeometryToolkitCore.Cameras.CameraModelVisualGeometryToolkitCore.Cameras.CameraModelVisualGeometryToolkitCore.Cameras.CameraModelVisualGeometryToolkitCore.Cameras.CameraModelVisualGeometryToolkitCore.Cameras.CameraModelVisualGeometryToolkitCore.Cameras.CameraModelVisualGeometryToolkitCore.Cameras.CanvasVisualGeometryToolkitCore.Cameras.CanvasTextureVisualGeometryToolkitCore.Cameras.OrientedCameraVisualGeometryToolkitCore.Cameras.PosedCameraVisualGeometryToolkitCore.Cameras.PosedCameraVisualGeometryToolkitCore.Cameras.PosedCameraVisualGeometryToolkitCore.Cameras.PosedPinholeCameraVisualGeometryToolkitCore.Cameras.angle_of_viewVisualGeometryToolkitCore.Cameras.backproject_with_jacobianVisualGeometryToolkitCore.Cameras.backproject_with_jacobianVisualGeometryToolkitCore.Cameras.bearing_correspondencesVisualGeometryToolkitCore.Cameras.calibration_matrixVisualGeometryToolkitCore.Cameras.calibration_matrixVisualGeometryToolkitCore.Cameras.default_frustum_depthVisualGeometryToolkitCore.Cameras.distortion_mapVisualGeometryToolkitCore.Cameras.distortion_mapVisualGeometryToolkitCore.Cameras.division_boundedVisualGeometryToolkitCore.Cameras.division_boundedVisualGeometryToolkitCore.Cameras.field_of_viewVisualGeometryToolkitCore.Cameras.fov_radiusVisualGeometryToolkitCore.Cameras.horizon_radiusVisualGeometryToolkitCore.Cameras.is_in_frontVisualGeometryToolkitCore.Cameras.is_in_frontVisualGeometryToolkitCore.Cameras.lens_coverageVisualGeometryToolkitCore.Cameras.lens_coverageVisualGeometryToolkitCore.Cameras.lookatVisualGeometryToolkitCore.Cameras.lookatVisualGeometryToolkitCore.Cameras.lookat_poseVisualGeometryToolkitCore.Cameras.metric_focal_lengthVisualGeometryToolkitCore.Cameras.normalized_pointVisualGeometryToolkitCore.Cameras.pinholeVisualGeometryToolkitCore.Cameras.plane_mapVisualGeometryToolkitCore.Cameras.radial_homographyVisualGeometryToolkitCore.Cameras.radial_homographyVisualGeometryToolkitCore.Cameras.radial_homographyVisualGeometryToolkitCore.Cameras.reframed_calibrationVisualGeometryToolkitCore.Cameras.reprojection_errorVisualGeometryToolkitCore.Cameras.sample_p3pVisualGeometryToolkitCore.Cameras.sample_p3pVisualGeometryToolkitCore.Cameras.sample_p3pVisualGeometryToolkitCore.Cameras.sensorVisualGeometryToolkitCore.Cameras.undistortVisualGeometryToolkitCore.Cameras.undistortVisualGeometryToolkitCore.Cameras.undistort!VisualGeometryToolkitCore.Cameras.undistort!VisualGeometryToolkitCore.Cameras.undistorted_rectVisualGeometryToolkitCore.Cameras.valid_rectVisualGeometryToolkitCore.Cameras.view_overlapVisualGeometryToolkitCore.Cameras.with_directionsVisualGeometryToolkitCore.Cameras.with_lens_coverageVisualGeometryToolkitCore.Cameras.with_presentVisualGeometryToolkitCore.backprojectVisualGeometryToolkitCore.backprojectVisualGeometryToolkitCore.backprojectVisualGeometryToolkitCore.backprojectVisualGeometryToolkitCore.extentVisualGeometryToolkitCore.extrinsicsVisualGeometryToolkitCore.focal_lengthVisualGeometryToolkitCore.focal_lengthVisualGeometryToolkitCore.focal_lengthVisualGeometryToolkitCore.modelVisualGeometryToolkitCore.p3pVisualGeometryToolkitCore.poseVisualGeometryToolkitCore.projectVisualGeometryToolkitCore.projectVisualGeometryToolkitCore.projectVisualGeometryToolkitCore.projectVisualGeometryToolkitCore.projectVisualGeometryToolkitCore.projectVisualGeometryToolkitCore.unprojectVisualGeometryToolkitCore.unproject