Camera Models

VisualGeometryToolkitCore provides a composable camera system with physical units throughout.

Camera Model Components

A camera is assembled from three components:

  1. Intrinsics — the calibration matrix K (LogicalIntrinsics; pixel units)
  2. Lens model — a map between rays and normalized image points (Pinhole, any Lens or LiftLens; see Lens Families)
  3. Extrinsics — the camera pose in world coordinates, as a frame-labeled EuclideanMap (see Coordinate Frames)

CameraModel pairs intrinsics with a lens; PosedCamera pairs a model with a pose.

Every camera — a model, a posed camera, a BoundedCamera of either — is an AbstractCamera. Whether it carries a pose is the trait is_posed: false for a model, true for a PosedCamera, the wrapped camera's for a BoundedCamera. The posed-only verbs (pose, extrinsics, rig membership) refuse an unposed camera with an ArgumentError, and PosedCamera(model, X) refuses a model that is already posed or bounded — the bound goes outside the pose, BoundedCamera(PosedCamera(model, X), extent). The trait is shared with the calibration targets: a PosedTarget is posed, a PlanarTarget or PointTarget is not, and model, pose and extrinsics answer on a posed member of either kind.

Quick Example

using VisualGeometryToolkitCore

# Select a sensor from the built-in database
s = sensor("Sony", "IMX174")

# Focal length from field of view
f = focal_length(s, deg2rad(40.0); dimension=:horizontal)

# Principal point at the image center (Rect(s) is COLMAP convention)
pp = centroid(Rect(s)) .* px

# Camera model (intrinsics + lens)
model = CameraModel(f, s.pitch, pp)

# Posed camera (extrinsics)
camera = PosedCamera(model, lookat(SVector(0.0, 0.0, 500.0),
                              SVector(0.0, 0.0, 0.0),
                              SVector(0.0, -1.0, 0.0)))

A lens on a sensor

The camera of a datasheet — a lens of a stated focal length on a stated sensor — is one call: the principal point at the sensor's centre, the camera bounded by the sensor so its pitch stays with it, and its focal length as a length read back by metric_focal_length:

sensor = Sensor(Size2(320, 240), 20μm)
camera = BoundedCamera(8mm, sensor; lens = Lens(DoubleSphere(0.25, 0.4)))   # fx = fy = 400 px, principal point (160, 120)
metric_focal_length(camera)                                                 # 0.008 m

A principal point off the sensor's centre is the general form, BoundedCamera(CameraModel(f, sensor.pitch, pp; lens), sensor) (pp in px), as in the posed example above.

Intrinsics

LogicalIntrinsics stores the calibration matrix K (and its inverse) in pixel units:

sz = Size2(1920, 1080)
K = CameraCalibrationMat(1000.0, sz)                    # principal point at the image centre, (960, 540)
K == CameraCalibrationMat(1000.0, Rect(sz))             # the same: the centroid of the image rect
K = CameraCalibrationMat((1000.0, 1002.0), Point2(961.5, 538.0))   # (fx, fy) and an explicit principal point
intrinsics = LogicalIntrinsics(K)

A focal length and a principal point are pixels, plain numbers or px quantities (1000.0px); the image's centre is (W/2, H/2) in COLMAP's convention, the image spanning [0, W] × [0, H].

Physical quantities enter through the constructors — a focal length in mm plus a pixel pitch produce the equivalent logical intrinsics (CameraModel(f, pitch, pp)); physical scale lives with the Sensor, not redundantly in the intrinsics.

Built-in Sensor Database

s = sensor("Sony", "IMX174")      # look up by manufacturer and model
s.resolution                      # Size2 in px
s.pitch                           # Size2 in μm/px

See CMOS_SENSORS for the registry.

Lens Models

A lens model is a map between rays and normalized image points: project(lens, X) and backproject(lens, x_d), valid past 90° wherever the lens images. has_distortion classifies the models; distortion_frame names each model's defining ray frame (see The Framed Imaging Chain).

There are two kinds of lens, built from a handful of parts — see Lens Families:

Every other lens map — the perspective views, Jacobians, the radial profile, undistortion — is derived from that pair; see Back-projection and Undistortion.

Projection and Backprojection

u = project(camera, X_world)          # 3D world point → pixel
ray = backproject(model, u)           # pixel → unit ray (camera frame)
X = unproject(model, u, depth)        # pixel + depth → 3D point

project composes extrinsics with the model's imaging map; backproject and unproject are the sections of the projection, on the ray domain of the lens (no tan θ blow-up at the equator; NaN where the lens does not image, or where a Z-depth names no point). transfer moves depth-attributed points between views (see Stereo Rigs and Uncertainty).

The camera's ray ↦ pixel map is available in two forms: the frame-free imaging_transform, affine_part(K) ∘ lens with the lens applied to the ray (so it is project(model, X) for every ray), and the fully-typed imaging_chain with every junction frame-checked — see The Framed Imaging Chain.

Camera Pose

Camera poses are frame-labeled rigid maps: CameraPose (X_WC, camera-to-world) and CameraExtrinsics (X_CW, world-to-camera) are frame-pair aliases of EuclideanMap, and inv converts between them by swapping the labels. PosedCamera(model, pose) and PosedCamera(model, extrinsics) both work — the constructor normalizes to the extrinsics slot, so handing a pose where extrinsics are expected is unrepresentable.

extrinsics = lookat(eye, target, up)      # world → camera EuclideanMap
p = lookat_pose(eye, target, up)          # the inverse convention
camera = PosedCamera(model, extrinsics)
pose(camera)                              # CameraPose (X_WC)

P3P Pose Estimation

# 3D world points and 2D image points
csp = correspondences(X, u)               # StructArray of Pt3-to-Pt2 correspondences

result = p3p(model, csp)          # FixedModels{4, Extrinsics3}, empty when no solution
for ext in result                 # 0–4 Extrinsics3 solutions
    camera_i = PosedCamera(model, ext)
    # score by reprojection error ...
end

The solver currency is the labeled Extrinsics3 alias (world-to-camera, a dispatch fact). sample_p3p draws minimal samples; the P3P solver type plugs into the solver layer.

Homographies

For a camera viewing the plane z = 0, the plane-induced homography is minted with frame labels (TargetPlaneFrame → PixelFrame{:colmap}):

H = HomographyMap(HomographyMat(camera))
u = H(Point2(100.0, 50.0))        # board point → pixel
Q = H(circle_world)               # a Conic: conics transform contravariantly

See Also