Getting started
PanoramaStitching turns a sweep of frames from a camera that only rotated into the rotation of every frame and a panorama. The registration holds the camera's calibration fixed; when the calibration is not known, the same frames refine it with the rotations, with no target (Autocalibration below). Its lens may have any field of view, so a fisheye imaging past 90° off axis works the same as a pinhole. All geometry is on unit bearings (VisualGeometryToolkitCore's backproject), never on the z = 1 plane, and every residual is measured in the camera's pixels.
The pipeline has four steps:
register_rotations— the rotationR_kof every frame (d = R_k·f_k, a bearingf_kof framekto a world directiond; the first frame's camera frame is the common frame).sample_frames— the frames sampled at directions spread uniformly over the covered sphere (fibonacci_directions): per direction, every frame that sees it with its sensor point, its value and the vignetting there — the observations of a vignetting validation or fit.render_panorama— the panorama into a view (panorama_view: a cylindrical-family projection or a virtual camera) by the reverse warp of the frames, unblended;render_cube_maprenders the six faces of a cube map.fit_vignetting— the camera's vignetting from the frames alone (every pixel of every frame, at its own pixel centre, is the scene's radiance times V), andscore_seams, how well a vignetting makes overlapping frames agree.
Saving the results writes them to a file.
Steps 2 and 3 run on VisualGeometryToolkitCore's warping machinery — image_warp/image_warp!, image_sample! and pointwise!, KernelAbstractions kernels — on the backend that owns the frames: GPU arrays (CuArray) run on the device, host arrays run the same code on the CPU.
This page continues from the first example: its camera, frames and reg are used below, and every block runs after it, in order, as the test suite runs them (the installation is the same).
The inputs
- Frames: a
VectorofH × W × 3linear (not gamma-encoded) images in capture order. They may be binned relative to the calibration:frame_mapis their map onto the camera's pixels (a CoreSimilarityMap; a RAW's:bin2frames', VisualGeometryToolkitCore'sraw_frame_map(L)), andplane_mapseach colour plane's (bayer_plane_map(L, c): a Bayer binning's R and B sit ¼ px off the cell centre; by default every plane's is the frame's). - Camera: a VisualGeometryToolkitCore
BoundedCamera(a camera model and its sensor). A parametricCameraModelserves every step. A grid lens is aCameraModelwith aSplineLens; it projects through its forward spline, fitted by VisualGeometryToolkitEstimation'sfit_camera. The example's camera as a 40 × 40 grid, with its forward spline:
# Illustrative: VisualGeometryToolkitEstimation's fit_camera, one more package (pkg> add VisualGeometryToolkitEstimation)
using VisualGeometryToolkitCore: SplineLens, imaged_region, with_model, estimate
import VisualGeometryToolkitEstimation as VgtkEstimation
# the same sensor and lens coverage, the model a 40 × 40 SplineLens with its forward spline
grid = with_model(camera, estimate(VgtkEstimation.fit_camera(SplineLens, camera, imaged_region(camera); tolerance = 1e-2,
grid_size = (40, 40))))
# register_rotations(frames, grid; …), sample_frames(grid, …), render_panorama(grid, …)- Imaged region: the camera is a VisualGeometryToolkitCore
BoundedCamera(the model and its sensor), and itsimaged_region— the sensor ∩ its lens coverage (the image circle, aCircleorEllipsein camera pixels:BoundedCamera(model, extent; lens_coverage = Circle(…)); omitted, the whole sensor) ∩ its model's valid domain — is the part of the sensor every step reads. Features and samples outside it are ignored; there is no separate circle argument.
1. Registering the rotations
register_rotations composes the stack's estimators and does no estimation of its own:
- Features (
PanoramaStitching.detect_features): ASIFT detections inside the camera's imaged region lessdetector_margin_px, described by VisualGeometryToolkitFeatures' DeDoDe/LoMa descriptor (descriptor = :b), at mostmax_detectionsper frame.tone_percentilesets the exposure the detector sees;asift_tilts = [(1.0, 0.0)]is the untilted view only. - Pairs:
pairs = :allmatches every frame pair;pairs = :lag_overlaponly those at mostmax_lagapart in capture order, plus every pair past the lag whose frames share any view under the rotation averaging of the verified pairs (VisualGeometryToolkitCore'sview_overlapof the frames posed by those rotations: the solid angle both image), withmin_inliersdeciding which are kept; no threshold. Matches come from the LoMa matcher (matcher = :b128) and become bearing correspondences through the camera (PanoramaStitching.pair_bearings), plain detections, so a pair's residuals are in the camera's pixels. - Pairwise rotations: VisualGeometryToolkitEstimation's
find_rotation(RANSAC on bearings, its sampler seeded fromseed) verifies each pair; a pair with fewer thanmin_inliersinliers is dropped. - Rotation averaging: the verified pairs form VisualGeometryToolkitEstimation's
ViewGraph(each edge the pair's rotation, weighted by its inlier count); Estimation'sfit_rotation_averagingof it withaveraging_optimization_strategy(anIrls; by default Estimation's own) gives every frame's rotation. - Tracks:
chain_trackslinks the inlier matches into tracks across frames (the components of the match graph, Estimation'sViewGraphon the observations); inconsistent tracks (two detections of one frame) are dropped. A kept pair whose frames share at leastnear_duplicate_overlapof the smaller footprint (VisualGeometryToolkitCore'sview_overlapunder the averaged rotations; 0.90 by default, the owner's value;0turns the check off) is a near-duplicate: averaged but seeding no track (reg.near_duplicates), since two near-identical views match their repeated structure as well as their scene, and a wrong match through them chains far. - Bundle:
refine_rotationsrefines every rotation but the first, and one direction per track, minimizing the robust pixel reprojection error through the camera's forward map with NLLSsolver: the strategybundle_optimization_strategy(aTriggs) atscale = σ̂, its width the strategy's efficiency constant × σ̂, σ̂ the noise scale of the verified pairs pooled over the kept ones (reg.bundle_sigma, pixels), in one LM run. The bundle refines the rotations and the directions only: the camera's calibration is always held, and it takes no keyword that would free a camera parameter. Refining the camera too is a refine of its own,refine_camera_and_rotations(Autocalibration below). - Track check (
check_tracks = true): the observations whose residual exceeds 3.035 σ̂ (√χ²₂(0.99)) are removed and the bundle re-solved from its rotations, until a solve removes none (at most 10 solves);reg.track_checkholds each solve's removals, rms and inlier rms (reg.bundle.metadata.inlier_rms, the rms over the observations within the loss width).
Steps 1–3 are verify_pairs, whose VerifiedPairs hold everything the rest reads; register_rotations(pairs::VerifiedPairs, camera; frame_map, …) runs steps 4–6 from them, so a caller that keeps the verified pairs (the example keeps them as a keyed product, verified_pairs/<key>/verified_pairs.jld2) reruns the averaging and the bundle alone.
The example's call states every setting: each is a keyword without a default, so a run states its whole configuration (say = println prints one line per frame, pair and step). The result, a Registration, holds the rotations, the view graph, per-pair statistics, the tracks and the bundle's result:
import VisualGeometryToolkitCore as VgtkCore
reg.pairs[(1, 2)] # matches, inliers, rms (the inliers' rms Sampson distance, px) and rotation_deg
length(reg.tracks), reg.dropped # the tracks, and the inconsistent ones dropped
reg.bundle.rms # the bundle's rms reprojection error, pixels
VgtkCore.retcode(reg.bundle) # how the bundle ended: Core's ReturnCode (and stop_rule, the rule that fired)On the example's sweep the rotations come back well within 0.05° and the bundle's rms is below half a pixel.
Autocalibration
A camera that only rotates determines its own calibration from its frames, with no target (Hartley 1994; Agapito, Hayman and Reid 2001): the observations of one track must back-project to one direction in every frame that sees it, which constrains the camera's parameters as well as the rotations. refine_camera_and_rotations is that refine. From a start camera (a nominal lens, or an earlier calibration of the same one), start rotations and the tracks, it refines the camera's parameters jointly with every rotation and one direction per track. Its one gauge is the first frame's rotation, held exactly.
It is a refine of the stack's robust-estimation contract, so it never estimates a scale of its own:
optimization_strategyis the robust strategy: aTriggsstrategy on a loss shape (its loss as NLLSsolver's robustifier at one width; the defaultTriggs{WelschShape}at the shape's efficiency constant). Another strategy is anArgumentError.- A robust strategy needs exactly one of
scale = σ(the noise scale, pixels) orwidth = w(the loss width). Here the scale is the registration's pooled noise scale. Neither, or both, is anArgumentError. fixednames the camera parameters to hold, by name or by group (:kfor every lens coefficient; VisualGeometryToolkitCore'sfree_parameter_names). The default()frees every parameter but the skew, which is held by construction.
The start must be close: like every refine, it ends in the local minimum its start leads to. Only a parametric camera can be refined: a SplineLens camera is an ArgumentError. The tracks come from a registration run with the start camera, so rerun register_rotations with the refined camera to verify the pairs through it.
# Illustrative: VisualGeometryToolkitCore's Triggs strategy
using VisualGeometryToolkitCore: Triggs
# the tracks as frame => pixel observations (each detection's centre through the frames' map), and the registration's
# rotations as the start, keyed by frame
tracks = [[k => frame_map(SA[reg.features[k][kp].center...]) for (k, kp) in t] for t in reg.tracks]
start = Dict(enumerate(reg.rotations))
autocal = refine_camera_and_rotations(camera, start, tracks; optimization_strategy = Triggs{CauchyShape}(efficiency_constant = 2.0),
scale = reg.bundle_sigma, fixed = (:aspect,), max_iter = 100, ftol = 1e-10,
xtol = 1e-10)2. Sampling the frames at directions
fibonacci_directions spreads directions uniformly over the sphere (the golden-angle spiral: equal area per direction, no pole bias), here restricted to those within fov_deg of some frame's axis. sample_frames takes each into every frame (f = R_kᵀ·d), projects it to its exact sub-pixel sensor point u through the camera, samples the frame there bilinearly and evaluates the vignetting model continuously at the same point, vignetting(u, f):
saturated = [falses(size(f, 1), size(f, 2)) for f in frames] # mask clipped pixels here
D = fibonacci_directions(reg.rotations, 200_000; fov_deg = 89.0)
S = sample_frames(camera, frames, reg.rotations, D; saturated, frame_map, rim_margin_px = 12.0, fov_deg = 89.0,
vignetting = (u, f) -> 1.0)A frame sees a direction (S.seen[i, k]) when the ray is within fov_deg of the camera's z axis, its sensor point lies inside the camera's imaged region less rim_margin_px at its lens coverage's edge and 2 px inside the frame, and no saturated pixel enters its value. For every direction seen by several frames, S.values[i, k, c] / S.vignetting[i, k] must agree across k: the spread is the vignetting model's error, and I_k = L(d)·V(u_k) is the equation of a panorama vignetting fit.
3. Rendering the panorama
panorama_view builds the raster and its pixel → ray map; render_panorama warps every frame into it (Core's image_warp, one fused map camera ∘ R_kᵀ ∘ rays per frame), divides by the vignetting and keeps, at each pixel, the LAST frame that sees it — no gain, no seam, no blend. It renders the frames as they are: a saturated pixel keeps its clipped value (the masks are for observations — the sampling, the fit and the seam score — not for pictures):
cyl = panorama_view(VgtkCore.Cylindrical(), (-120.0, 120.0, -40.0, 40.0), 0.25) # columns longitude, rows tan(latitude)
pano = render_panorama(camera, frames, reg.rotations, cyl; vignetting = (u, f) -> 1.0, frame_map, rim_margin_px = 12.0,
fov_deg = 89.0)
pano.image # rows × columns × 3, NaN where no frame sees the pixel
pano.owner # the frame each pixel holds
pano.coverage # how many frames see each pixelThe cylindrical family — Core's Equirectangular() (rows the latitude), Cylindrical() (tan φ), CylindricalEqualArea() (sin φ: every pixel subtends the same solid angle) and Mercator() (asinh(tan φ), conformal) — shares the longitude columns. An azimuthal view needs no projection of its own: it is a virtual camera, a Core CameraModel with the lens of the view, aimed by a rotation:
using VisualGeometryToolkitCore: Lens, Perspective, Stereographic
using StaticArrays: SMatrix
# a 1920 × 1080 rectilinear view, and a 'little planet': a stereographic view looking down
rect = panorama_view(CameraModel(1000.0px, SA[960.0px, 540.0px]; lens = Lens(Perspective())), 1920, 1080)
down = SMatrix{3, 3}(1.0, 0, 0, 0, 0, -1, 0, 1, 0) # its axis along +y (down)
planet = panorama_view(CameraModel(400.0px, SA[1000.0px, 1000.0px]; lens = Lens(Stereographic())), 2000, 2000; rotation = down)render_cube_map renders the six 90° faces and their horizontal cross. To level the horizon, supply up = up_frame(pitch, roll) to panorama_view (the first frame's pitch and roll against the true horizon, degrees).
4. The vignetting from the frames
A rotating camera sees each scene direction at several sensor points, so its frames determine its vignetting up to one scale. fit_vignetting uses every pixel of every frame inside the domain, read at its own pixel centre — no interpolation of the frames: log I_k(p) = log L(R_k·f(p)) + log V(u(p)), the pixel's ray f(p) the camera's back-projection (computed once: every frame shares the pixel grid), the scene log L a cubic B-spline over an AzimuthalChart of the covered sphere with a difference penalty, and V VisualGeometryToolkitCore's LogVignetting: V = exp(s), s a B-spline of log V in the distance from a centre (RadialVignetting) or over the sensor (TensorProductVignetting). The model is linear in log L, log V and the log gains, so the fit is one weighted linear least squares; its gauge is mean log V = 0 over the domain. Each row is the implicit constraint of its observation (sensor point, log I), weighted by its whitening by its noise: the value's — each frame's own variance where its image carries one (a LinearRaw under a sensor model whose noise feeds it: weight I²/var(I)), else the read-noise fallback of a plain image (weight I², these frames') — and, with observation_noise = PositionNoise(…), the pixel's sensor position too (the sensor point's from the registration's residual scale, at the first solve's model, so a row where the scene's log I changes fast across the sensor counts less; it needs the frames' variances). gain_correction = IntensityGains() adds an exposure gain per frame shared by every colour plane, I_k = g_k·L·V (gauge g₁ = 1), and ChannelGains() one per frame and colour plane, I_kc = g_kc·L_c·V (a white balance or a light colour that drifts too; V stays one for all planes): a fixed exposure under changing light needs them, or V absorbs the frames' differences. nothing takes the frames at one exposure. free_directions says how V's control points may move: a radial profile's pole-regular subspace (smooth at its centre), every control point of a tensor surface, or none (zeros(n, 0)) to hold a given V fixed, a calibration's — the same solve is then the frames' gain compensation. The normal equations are accumulated by atomic scatter-add on the frames' backend (GPU arrays on the device) and solved by conjugate gradients there, or by a host Cholesky factorization:
s = VgtkCore.RadialVignetting(16, SA[500.0, 500.0], 490.0) # controls, centre, radius (camera pixels)
v0 = VgtkCore.LogVignetting(VgtkCore.with_control_points(s, zeros(16))) # log V ≡ 0
res = fit_vignetting(v0, camera, frames, reg.rotations; free_directions = VgtkCore.pole_regular_basis(v0),
gain_correction = ChannelGains(), saturated, frame_map, rim_margin_px = 12.0, saturation_margin_px = 4.0, fov_deg = 89.0, plane = 2,
chart_spacing_deg = 1.0, penalty_order = 3, penalty_weight = 0.01,
vignetting_penalty_weight = 1.0e-6,
solver = ConjugateGradient(; tolerance = 1.0e-9, max_iter = 40_000), # or SparseCholesky()
observation_noise = nothing)
# or observation_noise = PositionNoise(; point_covariance = VgtkCore.CovMat{2}(σ = σ_u)), on frames with variances: σ_u the registration's
# pooled noise scale reg.bundle_sigma (the bundle's width over its efficiency constant), px
V = VgtkCore.estimate(res) # the fitted Core model: the vignetting keyword of render_panorama
g = res.metadata.gains # the frames' exposure gains, K × 3: g_kc per frame and colour plane (g_1c = 1)A sweep about one axis does not observe a 2-D V's tilt across it (a field that varies only with the ray's height is unchanged by a pure yaw, so L absorbs it): fit the tensor space only to frames with pitch and roll diversity. score_seams measures the result where it matters, at the seams of the unblended panorama: per raster pixel measured by two frames or more, the median log ratio of the other frames to the shown one (on the fit's observations: a direction any frame sees saturated is excluded from every frame), and its rms and 1.4826·MAD:
seams = score_seams(camera, frames, reg.rotations, cyl; vignetting = V, gains = g, saturated, saturation_margin_px = 4.0, frame_map,
rim_margin_px = 12.0, fov_deg = 89.0, plane = 2, chunk = 200_000)
seams.rms, seams.mad # the spread of the log ratio over the n pixels two frames or more measure
corrected = render_panorama(camera, frames, reg.rotations, cyl; vignetting = V, gains = g, frame_map,
rim_margin_px = 12.0, fov_deg = 89.0) # the panorama with V and the frames' gains divided outThe example's frames carry no vignetting and one exposure, so here V comes back close to 1 and every gain close to 1.
5. Saving the results
The results — the camera, the frames' rotations, the vignetting and the frames' gains — are saved and loaded through VisualGeometryToolkitIO's vgio/panorama/v2.0 kind (save_panorama, load_panorama; IO 0.10 or later). Its rotations are a VisualGeometryToolkitCore Trajectory from the camera's frame to the world, d = R_k·f_k with the world the first frame's camera frame, and a zero translation (a length: the camera only rotates):
# Illustrative: VisualGeometryToolkitIO, one more package (pkg> add VisualGeometryToolkitIO)
import VisualGeometryToolkitIO as VgtkIO
using VisualGeometryToolkitCore: Trajectory, CameraFrame, WorldFrame, EuclideanMap, m
using StaticArrays: SVector
# each R an SMatrix{3, 3}: its translation a static zero length
rotations = Trajectory(CameraFrame, WorldFrame, Dict(k => EuclideanMap(R, zero(SVector{3, Float64}) .* m) for (k, R) in enumerate(reg.rotations)))
VgtkIO.save_panorama("panorama.json", camera, rotations, V, g, reg.bundle) # g = nothing: no gains; the bundle's termination record (optional)
p = VgtkIO.load_panorama("panorama.json") # (; camera, rotations, vignetting, gains, fit)
Rs = [p.rotations[k].R for k in 1:length(p.rotations)] # the rotations render_panorama and score_seams take