API

Registration

PanoramaStitching.register_rotations — Function
register_rotations(frames, camera::BoundedCamera; frame_map, tone_percentile, detector_margin_px, asift_tilts, max_detections,
                   selection = :spread, detection_image = :tone, sensor_models = nothing, noise_sigmas = nothing, saturated = nothing,
                   descriptor, matcher, min_score, pairs, max_lag = nothing, min_inliers, seed,
                   scoring = nothing, local_optimization_strategy = nothing, max_trials, min_trials,
                   averaging_optimization_strategy = <Estimation's>, cycle_tol = deg2rad(2), orientation_tol = deg2rad(5),
                   bundle_optimization_strategy = Triggs{WelschShape}(…), bundle_max_iter, bundle_ftol,
                   bundle_xtol, near_duplicate_overlap = 0.90, check_tracks = true, execution_provider = :auto,
                   say = _ -> nothing) -> Registration
register_rotations(pairs::VerifiedPairs, camera::BoundedCamera; frame_map, averaging_optimization_strategy = <Estimation's>,
                   cycle_tol = deg2rad(2), orientation_tol = deg2rad(5), bundle_optimization_strategy = Triggs{WelschShape}(…),
                   bundle_max_iter, bundle_ftol, bundle_xtol, near_duplicate_overlap = 0.90, check_tracks = true,
                   say = _ -> nothing) -> Registration

The rotations of the pure-rotation sequence frames (H × W × 3 linear images in capture order, frame_map their map onto the camera's pixels: VisualGeometryToolkitCore's raw_frame_map) seen by the calibrated camera (fixed; a VisualGeometryToolkitCore BoundedCamera, whose imaged_region bounds the features, projected in the bundle through its model's imaging_transform — a grid lens's SplineLens through its forward spline; see refine_rotations), with the view graph, the tracks and the bundle's statistics. Every setting is the caller's configuration, none defaulted but the RANSAC's:

  • features: inside the camera's imaged_region; tone_percentile, detector_margin_px, asift_tilts (a Features tilt_schedule name, :imas25 or :untilted, or its pairs), max_detections, selection, detection_image, sensor_models (with :vst: one VisualGeometryToolkitCore SensorModel per frame, its noise a PoissonGaussian; nothing, the absent model of every frame) and noise_sigmas, saturated (the frames' clip masks, nothing: a LinearRaw's own, none for an array) — see detect_features (detection_image = :vst: the frames are LinearRaws); descriptor and matcher, VisualGeometryToolkitFeatures' LoMaDescriptor and LoMaMatcher variants (:b, :b128), one ONNX session each for the whole registration on ONNX Runtime's execution_provider (:auto, the default: CUDA where VisualGeometryToolkitFeatures finds a device and the environment holds CUDA_Runtime_jll and CUDNN_jll, else the CPU with a warning naming why; :cpu; :cuda, an error naming why where it cannot run), on Threads.nthreads() ONNX Runtime threads; min_score. The provider used is in the result's loma.
  • pairs: pairs = :all (every frame pair) or :lag_overlap (pairs at most max_lag apart; while the verified pairs leave the frames in more than one part, as when a lag pair is dropped, the pair between two parts nearest in capture order; then every pair whose frames share any view under the rotation averaging of the verified pairs — VisualGeometryToolkitCore's view_overlap of the frames posed by those rotations, a solid angle above zero — with min_inliers deciding which are kept; max_lag is given with :lag_overlap only, refused with :all); a pair is verified by find_rotation (its scoring and local_optimization_strategy objects, nothing for VisualGeometryToolkitEstimation's defaults; max_trials, min_trials; the sampler seeded from seed) and kept with at least min_inliers inliers. Its residuals are in the camera's pixels (pair_bearings: plain detections).
  • graph: VisualGeometryToolkitEstimation's fit_filtered_rotation_averaging of the verified pairs' view graph, its strategy averaging_optimization_strategy (an Irls{Shape}; by default VisualGeometryToolkitEstimation's [fit_rotation_averaging] optimization_strategy): the pairs off a cycle by more than cycle_tol removed (filter_cycles), the averaging, then the pairs off the averaged rotations by more than orientation_tol removed (filter_by_orientation) and the averaging again (radians; nothing skips that filter). A periodic scene verifies a pair one period off as well as the true one, and only the view graph tells them apart. The pairs the filters removed are the result's removed (a pair with no determined rotation covariance is never an edge: it is the result's no_covariance, not removed); their matches make no track.
  • scale: per verified pair its RANSAC scorer's own scale ŝ (Core's noise_scale of its result: the consensus set's Student-t posterior scale) and the result's degrees of freedom, the pairs' scales (VerifiedPairs), px. No second scale estimate.
  • bundle: refine_rotations with bundle_optimization_strategy (a Triggs{Shape}, by default VisualGeometryToolkitEstimation's pixel bundle's) at scale the kept pairs' scales pooled by VisualGeometryToolkitEstimation's pooled_noise_scale (σ̂² = Σ νₚŝₚ² / Σ νₚ, νₚ the pair's degrees of freedom; the bundle records it as ScaleUpstream, its width the strategy's efficiency constant × σ̂), σ̂ the result's bundle_sigma; ONE LM run (bundle_max_iter, bundle_ftol, bundle_xtol: refine_rotations), the width fixed. No kept pair with a scale is an ArgumentError. Every observation is its detection's centre in the camera's pixels (through the frames' frame_map), the residuals in pixels. The bundle holds the camera's calibration (refine_rotations); the retired bundle_camera_free, bundle_scale_multiple, the whitening switch and local_optimization are an ArgumentError naming their successors (the autocalibration of a rotating camera is refine_camera_and_rotations, on the result's rotations and tracks).
  • near-duplicates: a kept pair whose frames share at least near_duplicate_overlap of the smaller footprint — the solid angle both image, VisualGeometryToolkitCore's view_overlap under the averaged rotations, over the smaller frame's own (max(fraction_i, fraction_j)): PanoramaStitching's one overlap estimate, as :lag_overlap's — verifies and is averaged but seeds no track: two near-identical views match their repeated structure as well as their scene, and a wrong match through them chains far. The result's near_duplicates. Always on: near_duplicate_overlap defaults to 0.90 (the owner's value, owner ruling 2026-11-14); 0 turns the check off (every kept pair chained); a value outside [0, 1] is an ArgumentError. The camera's footprint is its imaged_region (its lens coverage), and view_overlap's refusals apply (a grid model that is not a CameraModel, a field of view within 3° of the backward axis).
  • track check (check_tracks = true): after the bundle, the observations whose residual exceeds 3.035 σ̂ (σ̂ the pooled scale above; √χ²₂(0.99): 1 % of the inliers at the true scale) are removed — a track left in fewer than two frames with them — and the bundle re-solved from its rotations, until a solve leaves none to remove or 10 solves have removed some (COLMAP's filter_max_reproj_error loop, Hugin's cpclean). The result's track_check holds each tested solve's removals, plain rms and inlier rms.

say receives one line per frame, pair and step.

The second form starts from the verified pairs (verify_pairs, the first form's first step: everything up to the view graph) and their frames' frame_map, so a caller that keeps them reruns the averaging and the bundle alone; the first form is register_rotations(verify_pairs(frames, camera; frame_map, …), camera; frame_map, …).

A task verb: it composes the estimators above and does none of its own. Each pair's RANSAC sampler is seeded from seed and the pair's frame indices, a stream of its own per (seed, i, j). A pairs other than :all or :lag_overlap, a max_lag below 1 or none with :lag_overlap or one given with :all, a near_duplicate_overlap outside [0, 1], the retired overlap_deg and near_duplicate_deg (PanoramaStitching 0.7's optical-axis angles, refused by name naming their successors), and an execution_provider other than :auto, :cpu or :cuda, and a camera whose footprint view_overlap refuses wherever the registration reads it (the near-duplicates, unless near_duplicate_overlap = 0; :lag_overlap's pairs past the lag) are an ArgumentError before any feature is detected. With either pairs, a view graph that is not connected once the pairs with fewer than min_inliers inliers are dropped (its frames have no common frame; VisualGeometryToolkitEstimation's require_connected) is an ArgumentError too.

source
PanoramaStitching.Registration — Type
Registration{G, A, B, F}

The result of register_rotations: rotations (one RotMatrix{3} per frame, d = R_k·f_k, the first frame's camera frame the common frame), graph (VisualGeometryToolkitEstimation's ViewGraph of the verified pairs the view-graph filters kept: per pair (i, j) VisualGeometryToolkitCore's Attributed(Uncertain(R_ij, CovMat(Σ_R)), (; weight)), f_j = R_ij·f_i, ΣR its absolute rotation covariance (rad²; VisualGeometryToolkitEstimation's `uncertainrotation), its weight the inlier count),removed(the verified pairs the filters removed, sorted),nocovariance(the verified pairs whose rotation covariance is not determined: not edges, sorted),pairs(per verified pair(i, j), kept or removed: matches, inliers, the rms Sampson residual of its inliers in pixels and its rotation angle in degrees),averaging(the averaging fitted ongraph, VisualGeometryToolkitEstimation'sfitfilteredrotationaveraging: the edges' covariances carry the scale, so it records scale 1,ScaleUpstream; the edges sharing a detection are treated as independent, the first-order approximation),nearduplicates(the kept pairs whose frames share at leastnearduplicateoverlapof the smaller footprint, VisualGeometryToolkitCore'sviewoverlapunder the averaged rotations: averaged, not chained; sorted),tracks(of the kept pairs but the near-duplicates, as(frame, detection)observations, after the track check),dropped(inconsistent tracks dropped by [chaintracks](@ref)),trackcheck(one entry per solve the track check tested: the observations itremovedpast the cut and that solve'srmsandinlierrms, px; empty withchecktracks = false),bundle(therefinerotationsresult; itsrmsover every kept observation and its metadata'sinlierrmsover those within its loss width; itsretcode/stoprule(Core's termination record) and its metadata'sndroppedstartsay how the bundle ended and how many observations it left out at its start, [RotationBundleMetadata](@ref)),bundlesigma(the pooled noise scale σ̂ of the kept pairs, px, the bundle'sscale),features(the frames' described features, [detectfeatures](@ref): a track's(frame, detection)is detectiondetectionoffeatures[frame], in the frame's pixels) andloma(what ran LoMa: VisualGeometryToolkitFeatures'lomaprovenance` of the descriptor and of the matcher — the models, the ONNX Runtime provider used and its threads).

source
PanoramaStitching.verify_pairs — Function
verify_pairs(frames, camera::BoundedCamera; frame_map, tone_percentile, detector_margin_px, asift_tilts, max_detections,
             selection = :spread, detection_image = :tone, sensor_models = nothing, noise_sigmas = nothing, saturated = nothing, descriptor, matcher, min_score, pairs, max_lag = nothing, min_inliers, seed,
             scoring = nothing, local_optimization_strategy = nothing, max_trials, min_trials, averaging_optimization_strategy = <Estimation's>,
             execution_provider = :auto, say = _ -> nothing) -> VerifiedPairs

The verified frame pairs of the pure-rotation sequence frames: the features, the matching and the pairwise rotations of register_rotations (its keywords of the same names), stopping before the view graph. max_lag and averaging_optimization_strategy are read only by pairs = :lag_overlap (its rotation averaging of the verified pairs aims the pairs past the lag), which needs max_lag; giving either with pairs = :all is an ArgumentError. The retired overlap_deg is refused by name (an ArgumentError: :lag_overlap tries every pair past the lag that shares any view). A pairs other than :all or :lag_overlap, a max_lag below 1 with :lag_overlap, a detector setting detect_features refuses (detection_image = :vst needs every frame a LinearRaw), an execution_provider other than :auto, :cpu or :cuda and, with :lag_overlap and a pair past the lag, a camera whose footprint VisualGeometryToolkitCore's view_overlap refuses (a model that is not a CameraModel, a region the lens cannot back-project, as a bare sensor's corners past a fisheye's fold, a field of view within 3° of the backward axis: give the camera its lens coverage) are an ArgumentError before any feature is detected.

source
PanoramaStitching.VerifiedPairs — Type
VerifiedPairs{F}

The verified frame pairs of a sequence, verify_pairs's result and the input of the rotations (register_rotations): features (per frame its described features, detect_features: VisualGeometryToolkitFeatures' StructArray{DescribedFeature}, in the frame's pixels), verified (per verified pair (i, j) its rotation R_ij, f_j = R_ij·f_i, its inlier count and its absolute rotation covariance (rad², the so(3) chart: uncertain_rotation; nothing when not determined): what the view graph and the rotation averaging consume), inlier_matches (per verified pair its inliers as detection index pairs (a, b), detection a of frame i ↔ detection b of frame j: what the tracks consume) and stats (per verified pair: matches, inliers, the rms Sampson residual of its inliers in pixels and its rotation angle in degrees), scales (per verified pair its noise scale: sigma, px per coordinate of its Sampson residual, its RANSAC scorer's own (NaN when the result recorded none), and dof, the result's degrees of freedom — see register_rotations) and loma (what ran LoMa: VisualGeometryToolkitFeatures' loma_provenance of the descriptor and of the matcher, (; descriptor, matcher)). Everything a registration needs after the features, the matching and the verification, so a caller may keep it and rerun the averaging and the bundle alone.

source
PanoramaStitching.chain_tracks — Function
chain_tracks(matches) -> (tracks, n_dropped)

The tracks of a sequence: the connected sets of observations (frame, detection) through every pair's inlier matches, matches an AbstractDict keyed by frame pairs (i, j) whose value is a vector of detection index pairs (a, b) (detection a of frame i ↔ detection b of frame j). The connected components of the match graph, VisualGeometryToolkitEstimation's ViewGraph on the observations (components). A track that holds two different detections of one frame is inconsistent (a wrong match chained through the graph) and is DROPPED whole; their count is n_dropped. Every track returned has observations in at least two frames, sorted by frame then detection; the tracks are sorted by their first observation.

source
PanoramaStitching.refine_rotations — Function
refine_rotations(camera, rotations, tracks; optimization_strategy = Triggs{WelschShape}(…), scale | width, max_iter,
                 ftol, xtol, eliminate = <NLLSsolver's Schur system>) -> FitResult{Dict{K, RotMatrix{3, Float64}}, RotationBundleMetadata}

The rotation bundle of a pure-rotation sequence seen by the calibrated camera (fixed: the calibration is held, a rotation-only bundle with no parameter keyword — refine_camera_and_rotations is the autocalibration; its backproject and backproject_with_jacobian give the start and the Jacobian; a CameraModel or a BoundedCamera), projected through its closed-form forward map π, its model's imaging_transform (a grid lens's SplineLens through its forward spline: give a SplineLens camera one with VisualGeometryToolkitEstimation's fit_forward_spline(camera; tolerance), which keeps its backward, K and frame, while fit_camera(SplineLens, …) turns a gauge-K camera onto its optical axis, a new frame). A residual is NaN off the camera model's valid_domain (a grid lens's lattice), where its back-projection Jacobian is undefined; the lens coverage does not enter the bundle. The rotations R_k (d = R_k·f_k) of every frame but the first of the sorted keys of rotations (the gauge, held fixed) and one direction d_j per track minimizing the robust sum of the pixel reprojection errors π(R_kᵀ d_j) − x_jk of every observation. A refine: it starts from rotations (a rotation averaging of the view graph, VisualGeometryToolkitEstimation.fit_rotation_averaging) and each track's mean bearing in the common frame.

An observation whose start prediction is not imaged (its residual or Jacobian not finite at the start: the prediction off the model's valid domain) is dropped, a track left in fewer than two frames is dropped whole (it keeps its start direction, held), and a partly kept track restarts from the mean bearing of the observations it keeps, its observations tested again until none is dropped: the solve is that of the kept observations alone. A frame other than the gauge left with no kept observation is held at its start rotation. The result's n is the number of observations kept; its metadata's n_observations the number given and n_dropped_start their difference (RotationBundleMetadata).

  • tracks — AbstractVector of tracks, each an AbstractVector of frame => pixel observations (pixels in the camera's frame), two frames at least. A pixel may be VisualGeometryToolkitCore's Uncertain(pixel, CovMat(Σ)), its 2 × 2 covariance Σ = LLᵀ (finite, positive definite, up to one global scale): the residual is then whitened, L⁻¹(π(R_kᵀ d_j) − x_jk), and the loss acts on the whitened residual. Every observation carries a covariance or none does.
  • optimization_strategy — the robust strategy of the robust-estimation contract: a Triggs{K} (VisualGeometryToolkitCore's; the default VisualGeometryToolkitEstimation's pixel bundle's, configs/default.toml [refine_camera_and_poses]: Triggs{WelschShape} at the shape's efficiency constant for a 2-D residual), its loss K as NLLSsolver's robustifier (loss_robustifier) at ONE width fixed for the bundle's one NLLSsolver run; or nothing, the plain least squares (no scale or width).
  • exactly one of scale = σ (the observations' noise scale, pixels or whitened units, or an upstream robust result: register_rotations hands the verified pairs' RANSAC scales pooled by Estimation's pooled_noise_scale) — the width is the strategy's efficiency constant × σ — or width = w (the loss width). Neither, or both, is an ArgumentError. max_iter, ftol (the relative cost decrease) and xtol (the largest step in chart units: radians for a rotation or a direction) are its stops (Core's LmStops, its one LM adapter lmsolve_chart! on an LmProblem); the result's iterations the LM iterations it took.
  • eliminate — whether NLLSsolver eliminates the directions (their Schur complement), which needs the loaded NLLSsolver's Schur system (MultiVariateLSschur; Core's nlls_features().schur): the default where it has one; false solves every unknown together, the default otherwise. eliminate = true without the Schur system is an ArgumentError naming it.

A refine descends towards the local minimum its start leads to, until a stop; the result's termination record (Core's retcode and stop_rule: a ReturnCode.T and a StopRule; converged(result) its success) says how the bundle ended: Core's lmsolve_chart! verdict, returned as it is (an ftol or xtol stop is Success with no stationarity test, so a loose tolerance converges short of the minimum; a step-size stop at a non-finite step is Stalled with NonFinite; the iteration cap MaxIters); max_iter = 0 (no solve: the start's rotations are the result) MaxIters with IterationCap. A track observed in fewer than two frames, or naming a frame without a start rotation, some observations with a covariance and some without, a covariance that is not finite and positive definite, and a start that keeps no observation (every track left in fewer than two imaged frames) are an ArgumentError, and so are the retired loss/loss_width (the strategy and its scale input replace them) and camera_free (the bundle holds the calibration: refine_camera_and_rotations frees it).

source
PanoramaStitching.refine_camera_and_rotations — Function
refine_camera_and_rotations(camera, rotations, tracks; optimization_strategy = Triggs{WelschShape}(…), scale | width,
                            fixed = (), max_iter, ftol, xtol, eliminate = <NLLSsolver's Schur system>)
    -> FitResult{Dict{K, RotMatrix{3, Float64}}, RotationBundleMetadata}

The AUTOCALIBRATION of a rotating camera: no target, a pure-rotation sequence (Hartley 1994; Agapito, Hayman and Reid 2001). The rotation bundle of refine_rotations — the same observations, start, kept set, gauge and result — with the camera's free parameters one more variable: VisualGeometryToolkitCore's ParameterManifold of the camera's model restricted to them (the camera block of Estimation's refine_camera_and_poses), its Jacobian ForwardDiff through the chart. The only gauge is the first frame's rotation, held exactly; with it a rotating sequence leaves nothing else free, which the metadata's calibration shows (the reduced information, and the parameters' covariance: RotationBundleMetadata). The observations kept at the start are those the GIVEN camera images; the residuals, the loss width and the result are the refined camera's.

A contract refine:

  • optimization_strategy — the robust strategy, a Triggs{K} (VisualGeometryToolkitCore's; the default VisualGeometryToolkitEstimation's pixel bundle's, configs/default.toml [refine_camera_and_poses]: Triggs{WelschShape} at the shape's efficiency constant for a 2-D residual): the loss K as NLLSsolver's robustifier at ONE fixed width.
  • exactly one of scale = σ (the observations' noise scale, px or whitened units; or an upstream robust result) — the width is then the strategy's efficiency constant × σ — or width = w (the loss width itself).
  • fixed — the camera parameters HELD, by name or by a coefficient family's group (:k for k1…, :d, :p, :sx, :sy): VisualGeometryToolkitCore's free_parameter_names(model, fixed). The default () frees every parameter but the skew, which the stack holds by construction (named in fixed it is a no-op). An unknown or repeated entry, a camera with nothing free, a SplineLens camera (its controls have no static chart), and the retired camera_free, free, fix, fixed_params and hold are an ArgumentError.
  • max_iter, ftol, xtol, eliminate — as refine_rotations's.
source
PanoramaStitching.RotationBundleMetadata — Type
RotationBundleMetadata

FitResult metadata of refine_rotations and refine_camera_and_rotations: directions (one unit direction per track, the common frame), residuals (per track, the pixel reprojection error norm of each of its observations at the result, unwhitened), loss_width (the width the strategy ran at: px, or in whitened units when the observations carry covariances; NaN for the plain least squares), inlier_rms (the rms of the residuals within loss_width — the loss's own inliers, the result's rms taken over them alone, beside the plain rms over every kept observation; NaN for the plain least squares or with no residual within it), scale and scale_source (the scale record of the robust-estimation contract: the given scale, or NaN with ScaleNone for a width or the least squares; read through VisualGeometryToolkitCore's noise_scale/scale_source), n_observations (given), n_dropped_start (the observations left out of the solve: those whose residual or Jacobian is not finite at the start — a start prediction off the camera model's valid domain, where it has no projection and no back-projection Jacobian, as for a grid lens calibrated over the sensor whose frames' observations reach its corners from a start a fraction of a degree off, the Tamron 20 mm of 2026-10-29 — and the rest of a track left in fewer than two frames; a dropped observation's residual is NaN, and a track left out keeps its start direction), frames_unconstrained (the frames other than the gauge left with no kept observation: each is held at its start rotation),

camera (the camera at the result: the one given by refine_rotations, which holds it, else the refined one, of the given camera's type and, for a BoundedCamera, its sensor and lens coverage) and calibration, nothing when the camera is held, else the autocalibration's information at the result (refine_camera_and_rotations):

  • free_parameters — the names of the camera parameters refined;
  • parameters, sigma, covariance — their values, standard deviations and covariance (parameter units: pixels for f, cx, cy; the lens's own for its coefficients) — s² × the camera block of the robust bundle's Gauss–Newton (JᵀJ)⁻¹ (each observation weighted by its loss's IRLS weight at the result, over the rotations, the track directions and the camera, the gauge frame removed; the directions eliminated by their Schur complement, the information accumulated block by block, never dense), its camera block mapped from the camera chart to the parameters; s² = Σ wᵢ‖rᵢ‖² / ν, ν = 2n − p (n observations kept, p unknowns: 3(K − 1) rotations, 2 per kept track, the free parameters);
  • s2, nu — s² and ν;
  • reduced_eigenvalues_gauge_fixed — the eigenvalues (ascending) of the information reduced to the rotations and the camera (the inverse of their marginal covariance: the Schur complement of the directions), the gauge frame removed, its rows and columns scaled to a unit diagonal — positive when nothing but the gauge is free;
  • direction_min_eigenvalue — the least eigenvalue over the kept tracks of their 2 × 2 direction blocks (each positive: every kept direction observed).
source
PanoramaStitching.refuse_retired_registration — Function
refuse_retired_registration(table, where::AbstractString) -> table

Refuse the registration's retired keys in table: a config's [registration] table (an AbstractDict with String keys, as TOML.parsefile gives it) or keywords (a NamedTuple or pairs(kwargs), Symbol keys). The first retired key found is an ArgumentError, "<where>:<key>is retired: <its successor, or how to migrate>"; otherwise table is returned as given (any other key is not this function's: register_rotations and verify_pairs refuse an unknown keyword themselves, after this check).

The ONE list of PanoramaStitching's retired registration settings, the keywords of register_rotations and verify_pairs and a config's keys alike: electrons_per_dn, read_noise_dn and detection_noise (the frames' sensor model), bundle_iterations, bundle_camera_free, bundle_scale_multiple, bundle_efficiency_constant and bundle_loss (bundle_optimization_strategy; refine_camera_and_rotations), averaging_loss (averaging_optimization_strategy), local_optimization (local_optimization_strategy), the whitening switch, the pairs' MSAC thresholds (the Student-t + uniform scoring), and 0.8's overlap_deg (:lag_overlap tries every pair that shares any view) and near_duplicate_deg (near_duplicate_overlap). A config loader calls it on the table it reads, before any stage, so a config fails the same way cached or not.

julia> using PanoramaStitching

julia> r = Dict{String, Any}("pairs" => "all", "near_duplicate_overlap" => 0.9);

julia> PanoramaStitching.refuse_retired_registration(r, "config.toml [registration]") === r
true

julia> try
           PanoramaStitching.refuse_retired_registration(Dict("near_duplicate_deg" => 4.0), "config.toml [registration]")
       catch e
           println(first(e.msg, 59))
       end
config.toml [registration]: `near_duplicate_deg` is retired
source

Features

PanoramaStitching.detect_features — Function
detect_features(camera::BoundedCamera, frame, descriptor; frame_map, tone_percentile, detector_margin_px, asift_tilts,
                max_detections, selection = :spread, detection_image = :tone, sensor_model = SensorModel(), noise_sigmas = nothing,
                saturated = nothing) -> StructArray{DescribedFeature}

The features of one frame (H × W × 3 linear, or a VisualGeometryToolkitCore LinearRaw of mode :bin2; frame_map its map onto the camera's pixels, Core's raw_frame_map(L)) of camera, described by descriptor (a LoMaDescriptor) on the frame's tone sqrt(G / p) clamped to [0, 1] (p the tone_percentile-th percentile of the green channel inside the camera's imaged_region — its sensor ∩ its lens coverage ∩ its model's valid domain), the detections in the frame's pixels (the descriptor resamples the image to its network's input).

  • ASIFT (asift_tilts: a VisualGeometryToolkitFeatures tilt_schedule name, :imas25 or :untilted, or its (t, φ) pairs) on the detection_image: :tone, that tone's green at Features' default contrast threshold; :vst, the frame's variance-stabilized green cells (Features' variance_stabilize(frame, sensor_model), noise σ = 1 at every level; the frame must be its LinearRaw, decoded with the same model, LinearRaw(mosaic; mode, sensor_model)) at noise_sigmas noise σ, sensor_model the frame's VisualGeometryToolkitCore SensorModel (its sensor_model resolver of the body's characterisation at the frame's ISO), whose noise must carry a gain (a PoissonGaussian; any other is refused naming the model's reason). With :tone neither is given (the absent SensorModel(), the default); the 0.6 keyword detection_noise is refused naming them.
  • No detection within three of its scales of a clipped pixel (Features' detection exclusion_mask): saturated, the frame's H × W saturation mask (a LinearRaw's own by default; nothing for an array frame: none).
  • The detections inside the imaged region less detector_margin_px at the lens coverage's edge (a camera whose coverage is its whole sensor: no margin).
  • selection: :spread (the one selection), about max_detections of the strongest spread over the frame (Features' spread_strongest, SSC, which the descriptor applies).

The result is the descriptor's own: VisualGeometryToolkitFeatures' described features (StructArray{DescribedFeature}), each a Detection in the frame's pixels (the detector's default: no detection covariance) and its DeDoDe description — the one feature store, in Features' canonical order of the detections (a function of the image: the same run to run, on any number of threads). frame_map takes a detection's centre to the camera's pixels (pair_bearings, the bundle of register_rotations).

source
PanoramaStitching.pair_bearings — Function
pair_bearings(camera, fa, fb, matches; frame_map) -> (cs, kept)

The matches of two frames' described features fa, fb (detect_features) as unit-bearing correspondences: VisualGeometryToolkitFeatures' correspondences of the matched features (their :score column kept), each detection's centre taken to the camera's pixels by the frames' frame_map and back-projected with its Jacobian as the M₁/M₂ columns (VisualGeometryToolkitCore's bearing_correspondences through the calibrated camera: the whole field, past 90°), plain points, so the pair's residuals are in camera pixels; and the detection index pairs kept (a pixel the camera cannot back-project is dropped).

source

Sampling the frames at directions

PanoramaStitching.world_direction — Function
world_direction(lat, lon) -> Vec3{Float64}

The unit direction of (latitude, longitude) in degrees: Core's Equirectangular back-projection (up = −y, longitude 0 = +z, +90° = +x).

source
PanoramaStitching.fibonacci_directions — Function
fibonacci_directions(n) -> Vector{SVector{3, Float64}}
fibonacci_directions(rotations, n; fov_deg) -> Vector{SVector{3, Float64}}

n unit directions spread uniformly over the sphere, the Fibonacci (golden angle) spiral: direction i at height 1 − (2i − 1)/n along the up axis (−y) and azimuth i·π(3 − √5) — equal area per direction, no pole bias. With rotations, the directions of the n that lie within fov_deg of some frame's optical axis (R_k·z): the sphere's set restricted to the frames' coverage, the input of sample_frames.

source
PanoramaStitching.FrameSamples — Type
FrameSamples

The frames sampled at a set of directions (sample_frames), N directions × K frames: directions (N, the common frame), pixels (N × K: the sensor point, camera pixels), rays (N × K: the ray in the frame's camera frame, R_kᵀ·d), values (N × K × 3: the frame's linear value there, NaN where unseen), vignetting (N × K: the vignetting model there, NaN where unseen) and seen (N × K). The arrays live on the backend of the inputs.

source
PanoramaStitching.sample_frames — Function
sample_frames(camera::BoundedCamera, frames, rotations, directions; saturated, frame_map, rim_margin_px, fov_deg,
              vignetting, plane_maps = ntuple(_ -> frame_map, 3)) -> FrameSamples

The frames sampled at directions (unit 3-vectors of the common frame, any array): for every frame k and direction d, the ray f = R_kᵀ·d (rotations[k] with d = R_k·f), its exact sub-pixel sensor point u = π(f) through the calibrated camera (a VisualGeometryToolkitCore BoundedCamera, projected through its model's imaging_transform), the frame's value there by bilinear sampling (Core's image_sample!, no smoothing) and the vignetting model evaluated continuously there, vignetting(u, f) (a model defined on the sensor reads u, one defined on the ray's direction reads f; a calibration's vignetting, or (u, f) -> 1.0 for none). These are the observations of a vignetting validation (I_k / V_k must agree across the frames that see a direction: their spread is the model's error; score_seams scores them).

A frame sees a direction by render_panorama's masks: the ray within fov_deg of the camera frame's z axis, the sensor point inside the camera's imaged_region (its sensor ∩ its lens coverage ∩ its model's valid domain) less rim_margin_px at its lens coverage's edge, and at least 2 px inside the frame, and no saturated pixel among those any plane's value blends (the four at each plane's own sample position). frames are H × W × 3 linear images, saturated their H × W masks. frame_map is the frames' map onto the camera's pixels (VisualGeometryToolkitCore's raw_frame_map(L)), and plane_maps each plane's (bayer_plane_map(L, c); by default every plane's is the frame's): plane c is read at inv(plane_maps[c])(u).

Every step is a batch on the backend that owns frames and directions (Core's pointwise!, image_sample!: one kernel each on a GPU array, the same code on the host), so for device arrays vignetting must be kernel-lawful (isbits, no allocation, no throw); it is evaluated only where the frame sees the direction. The precision is the directions' element type.

source
PanoramaStitching.frame_region — Function
frame_region(camera::BoundedCamera, width, height, rim_margin_px) -> region

Where a frame's sensor point counts, in camera pixels (a region answering p ∈ region): at least 2 px inside the width × height frame (camera pixels; the bilinear taps) and inside what camera images — its imaged_region, the sensor ∩ its lens coverage ∩ its model's valid domain — less rim_margin_px at its lens coverage's edge (the image circle's soft rim; the sensor's edge is not eroded, so a camera whose coverage is its whole sensor loses nothing to the margin). The domain of sample_frames, render_panorama, score_seams and fit_vignetting.

source

The panorama

PanoramaStitching.true_up — Function
true_up(pitch, roll) -> SVector{3}

The true up direction in the common (first-frame) frame: Rz(roll)·(0, −cos p, −sin p), p the first frame's pitch-down against the true horizon and roll its roll, in degrees. A level first frame's up is its camera frame's −y:

julia> using PanoramaStitching

julia> true_up(0, 0) == [0, -1, 0]
true
source
PanoramaStitching.up_frame — Function
up_frame(up::AbstractVector) -> SMatrix{3,3}
up_frame(pitch, roll) -> SMatrix{3,3}

R with d = R·d_true: the true-up frame of the up direction up (in the common frame; normalized here) — its −y up, its forward (+z) the common frame's forward (0, 0, 1) projected onto the horizontal, its x the right-handed completion — VisualGeometryToolkitCore's lookat_pose frame (the :cv convention) looking along that forward. up_frame(pitch, roll) is up_frame(true_up(pitch, roll)) (true_up, degrees); up may also be a sweep's own, sweep_up(rotations).up. The up keyword of panorama_view.

source
PanoramaStitching.sweep_up — Function
sweep_up(rotations) -> (; up, eigenvalues)

The up direction of a roll-free sweep from its frames' rotations alone (d = R_k·f_k, the common frame's directions): a roll-free frame's image x axis (R_k's first column) is horizontal, so up is orthogonal to every one — the eigenvector of the least eigenvalue of Σ_k x_k x_kᵀ — signed against the frames' summed image-down axes (R_k's second columns). eigenvalues are the matrix's, ascending: the least is ≈ 0 when the sweep is roll free (the diagnostic of the estimate). It holds whatever the frames' pitch, unlike the axis of the frames' rotations. Its frame is up_frame(sweep_up(R).up).

source
PanoramaStitching.PanoramaView — Type
PanoramaView

The raster a panorama is rendered into: rays, its pixel → ray map (a Transformation from the raster's COLMAP pixel (x, y) to the direction in the common frame), size (rows, columns) and extent ((lon₀, lon₁, lat₀, lat₁) degrees of a cylindrical-family raster, nothing for a virtual camera). Built by panorama_view.

source
PanoramaStitching.panorama_view — Function
panorama_view(projection, extent, deg_per_pixel; up = I) -> PanoramaView
panorama_view(camera, width, height; rotation = I) -> PanoramaView

The raster of a panorama and its pixel → ray map.

  • A cylindrical-family projection (Core's Equirectangular(), Cylindrical(), CylindricalEqualArea(), Mercator()): the raster spans extent = (lon₀, lon₁, lat₀, lat₁) degrees, deg_per_pixel along the longitude (its columns, from lon₀) and the same step in the projection's vertical coordinate (its rows, from lat₁ at the top: radians of latitude for the equirectangular, of tan φ, sin φ or asinh(tan φ) for the others), in the frame up (d = up·d_up, up_frame for a true-up panorama). Its pixel → ray map is LinearMap(up) ∘ inv(projection) ∘ raster, and its extent the raster's own (the latitudes of its row edges).
  • An azimuthal view, a virtual camera (a Core CameraModel): its width × height pixels, their rays LinearMap(rotation) ∘ Backprojection(camera) (d = rotation·f, f the virtual camera's ray). No new projection type: the lens chooses the view.
julia> using PanoramaStitching, StaticArrays

julia> using VisualGeometryToolkitCore: CameraModel, Lens, Perspective, Stereographic, Cylindrical, px

julia> cyl = panorama_view(Cylindrical(), (-120.0, 120.0, -40.0, 40.0), 0.05);   # cylindrical

julia> cyl.size                                     # rows in steps of tan φ, columns of longitude
(1923, 4800)

julia> rect = panorama_view(CameraModel(1000.0px, SA[960.0px, 540.0px]; lens = Lens(Perspective())), 1920, 1080);

julia> rect.size                                    # rectilinear
(1080, 1920)

julia> down = SMatrix{3, 3}(1.0, 0, 0, 0, 0, -1, 0, 1, 0);   # its axis along +y (down)

julia> planet = panorama_view(CameraModel(400.0px, SA[1000.0px, 1000.0px]; lens = Lens(Stereographic())), 2000, 2000;
                              rotation = down);    # a 'little planet'

julia> planet.size
(2000, 2000)
source
PanoramaStitching.render_panorama — Function
render_panorama(camera::BoundedCamera, frames, rotations, view; vignetting, frame_map, rim_margin_px, fov_deg,
                plane_maps = ntuple(_ -> frame_map, 3), gains = nothing) -> (; image, owner, coverage, extent)

The panorama of frames in the raster view (panorama_view) by the REVERSE warp, Core's image_warp/image_warp!: every pixel's ray d (the view's pixel → ray map) is rotated into each frame k (f = R_kᵀ·d, rotations[k] with d = R_k·f), projected through the calibrated camera (a VisualGeometryToolkitCore BoundedCamera, through its model's imaging_transform) to a sensor point between pixels, and sampled there bilinearly — one fused map camera ∘ R_kᵀ ∘ view.rays per frame — then divided by the vignetting vignetting(u, f), looked up at the sensor point u (camera pixels) and the ray f (the frame's camera frame) — a model defined on the sensor reads u, one defined on the ray's direction (a field angle) reads f; a Core AbstractVignetting reads u — and by the frame's exposure gain gains[k, c] of each colour plane c when gains is given (K × 3, per frame and plane: fit_vignetting's metadata.gains; nothing, the frames taken as one exposure). No seam, no blend, and each pixel keeps the value of the LAST frame that sees it, so a wrong vignetting shows as a step at every frame boundary.

A frame sees a pixel when its ray is within fov_deg of the camera frame's z axis, its sensor point lies inside the camera's imaged_region (its sensor ∩ its lens coverage ∩ its model's valid domain) less rim_margin_px at its lens coverage's edge, and at least 2 px inside the frame — the geometric masks of sample_frames, evaluated as batches at the warp's own sensor points. The render shows the frames as they are: a saturated pixel is rendered with its clipped value (a window that clips in every frame comes out bright, not empty); saturation masks observations only (sample_frames, fit_vignetting, score_seams). frames are H × W × 3 linear images. frame_map is the frames' map onto the camera's pixels (VisualGeometryToolkitCore's raw_frame_map(L)), and plane_maps each plane's (bayer_plane_map(L, c); by default every plane's is the frame's); plane c is read through inv(plane_maps[c]) ∘ camera ∘ R_kᵀ ∘ rays, all three planes in one Core image_warp!. The masks and the vignetting are at the pixel centre's sensor point.

Every step runs on the backend that owns frames (GPU arrays: one kernel per warp, mask and update; the same code on the host); for device arrays vignetting must be kernel-lawful (it is evaluated only where a frame sees the pixel). The warp's math — the sensor points, the rays, the bilinear weights — is in Float64 whatever the frames' element type (a Float32 sensor coordinate of a 4000 px sensor is 2.4e-4 px coarse, enough to move a sampled value by 1e-4 relative); the image is in the frames' element type.

Returns image (rows × columns × 3, NaN where no frame sees the pixel), owner (the frame whose value a pixel holds, 0 for none), coverage (how many frames see each pixel), all on the frames' backend, and the view's extent.

source
PanoramaStitching.render_cube_map — Function
render_cube_map(camera, frames, rotations, width; up = I, kwargs...) -> (; faces, cross)

The cube map of the panorama: six width × width 90° rectilinear views (a Lens(Perspective()) virtual camera of focal length width/2 px, principal point at the face's centre), each a render_panorama (kwargs are its keywords) of the panorama_view aimed by up·R_face. The faces, in the common frame up (x right, y down, z forward):

faceoptical axisimage rightimage down
front+z+x+y
right+x−z+y
back−z−x+y
left−x+z+y
up−y+x+z
down+y+x−z

faces is a NamedTuple of the six renders in that order; cross is the horizontal cross, 3·width × 4·width × 3 (NaN off the cross) — up above front, the row left, front, right, back, down below front — in which every shared edge of two faces is adjacent: the side faces' tops meet up's edges and their bottoms down's.

source
PanoramaStitching.coverage_extent — Function
coverage_extent(camera, frames, rotations, up, frame_map; δ = 0.5) -> (lon₀, lon₁, lat₀, lat₁)

The longitude and latitude box (degrees) the frames cover in the view of the up frame up (e.g. up_frame(sweep_up(rotations).up)): the seen pixels of a coarse equirectangular render_panorama at δ degrees per pixel over the whole sphere (latitudes within ±89°), no vignetting, the frames to their sensor's edge within 89° of the optical axis; the box of its seen pixels' edges. The extent of a panorama_view that frames the sweep.

source
PanoramaStitching.CUBE_FACES — Constant
CUBE_FACES

The six faces of a cube map, front, right, back, left, up, down, each the rotation R taking the face camera's ray f to the common frame (d = R·f): its columns the face's image right, image down and optical axis (see render_cube_map).

source

The vignetting from the frames

PanoramaStitching.fit_vignetting — Function
fit_vignetting(vignetting, camera::BoundedCamera, frames, rotations; free_directions, gain_correction, saturated, frame_map,
               rim_margin_px, saturation_margin_px, fov_deg, plane, plane_maps = ntuple(_ -> frame_map, 3), chart_spacing_deg, penalty_order, penalty_weight,
               vignetting_penalty_weight, solver, observation_noise)
    -> FitResult{<:VisualGeometryToolkitCore.AbstractVignetting, VignettingMetadata}

The vignetting V and the exposure gains of a camera that only rotated, from its frames alone (no flat field): every pixel of every frame is the scene's radiance in its direction times V at its sensor point times the frame's gain, I_k(p) = g_k·L(R_k·f(p))·V(u(p)), and V is the factor the frames share where their views of the scene overlap (self-calibration). No interpolation of the frames: each value is read at its own pixel centre, its ray f(p) the camera's back-projection (Core's batched backproject of the model of the VisualGeometryToolkitCore BoundedCamera, computed once for the pixel grid the frames share), and the scene L is a B-spline over an AzimuthalChart of the covered sphere whose values are fitted with V and the gains.

  • vignetting: V's model and start, a VisualGeometryToolkitCore LogVignetting (V = exp(s), s a RadialVignetting or TensorProductVignetting B-spline of log V on the sensor, camera pixels; its control points the start of log V, e.g. zeros); held fixed, any vignetting the render takes (a Core model, or a callable vignetting(u, f): a calibration's map, a flat field; differentiable in u for PositionNoise). A sweep about one axis cannot observe a free 2-D V's tilt across that axis (a field exp(α·f_y) is unchanged by a pure yaw, so L absorbs it): use a tensor surface for frames with pitch and roll diversity.
  • free_directions: the matrix Z (n_control_points × m) V's control points move along, vec(P) = vec(P₀) + Z y (Core's restrict of V's ParameterManifold to Z): Core's pole_regular_basis(v) for a radial profile (smooth at its centre), the identity for a tensor surface, zeros(n, 0) to hold the given V fixed — then the fit is gain compensation under that V (a calibration's), the same system without V's block.
  • gain_correction: nothing (every frame at one exposure, g = 1), IntensityGains() (one gain per frame, `log Ik = log gk + log L(d)
    • log V(u),K − 1unknowns with the gaugeg₁ = 1`: L carries the first
    frame's exposure) or ChannelGains() (one per frame and colour plane, the gauge g_1c = 1: each other plane's gains solved under the fitted V, which stays one for all planes). In both modes of V — fitted, or held fixed. Use gains when the exposure or the light can change between frames: a fixed shutter under a drifting sky changes every frame's level, and the fit then puts the difference into V (on the rooftop sweep the gains reached +18 % and V at the domain's edge came out 7 % too dark without them); per channel when the white balance or the light's colour can change too. Required, no default (TOML: GAIN_CORRECTIONS).
  • The fit domain: every pixel of the H × W × 3 frames whose sensor point lies inside the camera's imaged_region (its sensor ∩ its lens coverage ∩ its model's valid domain) less rim_margin_px at its lens coverage's edge, and 2 px inside the frame, with its ray within fov_deg of the optical axis. A row (a pixel of a frame) counts when its value is positive and its scene direction is saturated in NO frame whose domain sees it: each frame's saturated mask is dilated by saturation_margin_px camera pixels (≥ 0, ≈ the registration error, so a direction the rotations misplace by that much is still caught), and a row is dropped from every frame where any frame's dilated mask holds its direction (the camera's projection of R_jᵀ·R_k·f). A frame's own mask alone would drop a bright scene point where V is high (it clips there) and keep it where V dims it below white, so V would come out too flat at the rim. plane is the plane fitted (2: green), read at its own sample position. frame_map/plane_maps: the frames' and the planes' maps onto the camera's pixels (VisualGeometryToolkitCore's raw_frame_map, bayer_plane_map; by default every plane's is the frame's).
  • The model: log I = log g + log L(d) + log V(u), linear in log L's controls, log V's control points and the log gains: one weighted linear least squares (the weights normalized by their mean over the rows), with the difference penalty of order penalty_order (2 or 3) and weight penalty_weight (in units of one row of mean weight) on L's controls along both chart axes, chart_spacing_deg apart, a weak second-difference penalty on V's control points of weight vignetting_penalty_weight relative to the V block's mean diagonal (it fixes the control points the domain does not reach), and, with free directions, the gauge mean log V = 0 over the domain's pixels. A V held fixed enters as the numbers log V(u).
  • The value's noise, by dispatch on the frames' image type (all plain or all uncertain): an uncertain image — VisualGeometryToolkitCore's LinearRaw under a sensor model whose noise feeds each value's variance (a PoissonGaussian, or a specified RelativeNoise), its StructArray of Uncertain values — weights a row by I²/var(I); a plain image carries no variance and takes the read-noise FALLBACK, a constant σ in I, weight I² (the weights of every fit before 0.7). The metadata's value_variance says which.
  • observation_noise: the rest of a row's observation noise (required): nothing (the value's alone, one solve) or PositionNoise (the pixel's sensor position too: after that solve, iterations more, each with its weights from the previous solve's model; it needs the frames' variances). The row weights are always the whitening of the log-I constraint by the whole noise (Core's whitening_factor). TOML: OBSERVATION_NOISES.
  • The solve: solver, a ConjugateGradient (Jacobi-preconditioned, on the frames' backend) or a SparseCholesky (the same system assembled sparse and factored on the host).

Every stage runs on the backend that owns frames (GPU arrays: KernelAbstractions kernels and atomic scatter-add; host arrays, the same kernels on the CPU). The saturated masks live on the same backend.

Returns a FitResult whose estimate is the fitted Core vignetting model (isbits: the vignetting keyword of render_panorama, sample_frames and score_seams, on the host or a device; the given model when V is held fixed), its termination record (VisualGeometryToolkitCore's retcode, a ReturnCode.T, and stop_rule; converged(result) its success) the worst over every linear solve of every plane, the other planes' of ChannelGains included (Core's worst_retcode: a CG that ended at its budget in any solve is MaxIters), iterations the CG's summed over every solve of every plane (0 for Cholesky), rms the rms of the log residual over n rows weighted by the last solve's weights, and VignettingMetadata (its gains, the frames' K × 3 g_kc: the gains keyword of score_seams and render_panorama; the residuals and the solves are the fitted plane's).

source
PanoramaStitching.AzimuthalChart — Type
AzimuthalChart(axis, half_extent, spacing)

The chart of the scene's log L: VisualGeometryToolkitCore's azimuthal-equidistant direction chart about axis (chart, Core's equidistant_chart(axis): a direction at angle θ from the axis lands at radius θ, radians, in the direction of its azimuth; singular only at -axis), and the uniform cubic tensor B-spline of n × n controls laid over the closed square area of it, [−half_extent, −half_extent + (n − 3)·spacing]² with n = ⌈2·half_extent / spacing⌉ + 4, so the controls are spacing (radians) apart: Core's spline_window / spline_value lattice, the controls [row, column] along the chart's (second, first) coordinate. fit_vignetting centres it so that its singular point is as far from the frames' coverage as the sphere allows (for a cap-like coverage, close to the mean optical axis; for a full 360° sweep, the pole the frames do not see).

source
PanoramaStitching.VignettingMetadata — Type
VignettingMetadata

The metadata of fit_vignetting's FitResult: chart (the scene's AzimuthalChart), scene (log L's n × n controls, host, the chart's layout, so that VisualGeometryToolkitCore.spline_value(scene, chart.area, chart.chart(d)) is log L at the direction d; NaN where no observation reached a control), gains (K × 3: the frames' exposure gains g_kc of I_kc = g_kc·L_c·V, one per frame and colour plane, g_1c = 1; all 1 without gain correction, and one per frame — every column the same — with IntensityGains), the residual spread of log I of the fitted plane over the rows (residual_rms unweighted, residual_mad StatsBase's normalized mad, the 1.4826·MAD, residual_quantiles the 1 %, 50 % and 99 % quantiles; the result's rms is the one weighted by the last solve's weights), relative_residual (the last solve's ‖r‖/‖b‖), solver, value_variance (true when the frames' images carried each value's variance, the weights I²/var(I); false for plain images, the read-noise fallback, weights I²), observation_noise (nothing or the PositionNoise of the fit: the fitted plane's linear solves are 1, or 1 + iterations with a PositionNoise), log_change (the largest change of the model's log I over the domain's rows between the last two solves; 0 for one solve), n_pixels (the domain's pixels), n_active (scene controls with data) and seconds (the stages' wall times, summed over the solves). The fit's termination record is its result's (VisualGeometryToolkitCore's retcode, a ReturnCode.T, and stop_rule): the worst over every linear solve of every plane fitted (Core's worst_retcode) — a Cholesky solve Success with ClosedForm, a CG Success with ToleranceMet or, at its budget, MaxIters with IterationCap.

source
PanoramaStitching.ConjugateGradient — Type
ConjugateGradient(; tolerance, max_iter)

fit_vignetting's solve by Jacobi-preconditioned conjugate gradients on the frames' backend (VisualGeometryToolkitCore's normal_cg on the applied normal equations), to the relative residual tolerance (‖g − N x‖ ≤ tolerance·‖g‖, g the right-hand side) within max_iter iterations (its termination MaxIters with IterationCap when the budget ends first, Success with ToleranceMet otherwise: VisualGeometryToolkitCore's ReturnCode).

source
PanoramaStitching.PositionNoise — Type
PositionNoise(; point_covariance, iterations = 1)

fit_vignetting's observation_noise: the pixel's sensor position is uncertain too, beside its value's noise (the frames' own variance: an uncertain image, VisualGeometryToolkitCore's LinearRaw under a sensor model whose noise feeds one). A row is the implicit constraint

g(θ; x) = log I − log L(R_k·f(u)) − log V(u) − γ_k = 0

on the observation x = (u, log I), the pixel's sensor point u (camera pixels) and its log value, of covariance Σ_x = blockdiag(Σ_u, var(I)/I²), and its weight is the frozen whitening of that constraint by this noise, 1/(G_x·Σ_x·G_xᵀ) (VisualGeometryToolkitCore's whitening_factor of the constraint's constraint_with_jacobians), with the observation gradient G_x = [−∇_u(log L∘R_k∘f + log V)ᵀ, 1] of the MODEL (its scene spline and V) at the previous solve; the first solve is the value noise's alone.

  • point_covariance: Σ_u, VisualGeometryToolkitCore's CovMat{2} of a pixel's sensor point in camera pixels² (CovMat{2}(σ = σ_u) the isotropic σ_u²·I). A full or diagonal CovMat must be symmetric positive definite.
  • iterations: the reweight-and-solve cycles after the first solve, 1 or 2 (the second re-evaluates G_x at the first's solution).

A plain image (no variance) is refused with it. It replaces 0.6's value-and-position noise, whose constant σ of log I is now a specified RelativeNoise of the frames' sensor model; 0.6's value-only noise is the plain image's read-noise fallback.

source
PanoramaStitching.IntensityGains — Type
IntensityGains()

fit_vignetting's gain_correction: one exposure gain per frame shared by every colour plane (a gain on the intensity), log I_kc = log g_k + log L_c(d) + log V(u), K − 1 unknowns with the gauge g₁ = 1, fitted on the fitted plane and taken for every colour plane.

source
PanoramaStitching.ChannelGains — Type
ChannelGains()

fit_vignetting's gain_correction: one exposure gain per frame and colour plane, log I_kc = log g_kc + log L_c(d) + log V(u), the gauge g_1c = 1 per plane: a frame's exposure and its white balance or colour shift. V is one, shared by the planes: fitted (or held) on plane with that plane's gains, then each other plane's gains solved under that V (the same system with V held fixed, its own scene L_c).

source
PanoramaStitching.GAIN_CORRECTIONS — Constant
GAIN_CORRECTIONS

The TOML tags of fit_vignetting's gain_correction (a VisualGeometryToolkitCore TagRegistry, family suffix Gains): "none" (nothing: every frame at one exposure), "intensity" (IntensityGains) and "channel" (ChannelGains); the example's vignetting.gain_correction. A tag of an earlier release is refused with an ArgumentError that names its new tag (CHANGELOG, 0.4.0).

source
PanoramaStitching.OBSERVATION_NOISES — Constant
OBSERVATION_NOISES

The TOML tags of fit_vignetting's observation_noise (a VisualGeometryToolkitCore TagRegistry, family suffix Noise): "none" (nothing: the value's noise alone) and "position" (PositionNoise, whose settings the config states beside it); the example's vignetting.observation_noise. A tag of an earlier release is refused with an ArgumentError that names its new tag (CHANGELOG, 0.4.0 and 0.7.0).

source
PanoramaStitching.score_seams — Function
score_seams(camera::BoundedCamera, frames, rotations, view; vignetting, saturated, saturation_margin_px, frame_map, rim_margin_px,
            fov_deg, plane, plane_maps = ntuple(_ -> frame_map, 3), chunk, gains = nothing) -> (; ratio, rms, mad, n)

How well vignetting makes the frames agree where they overlap, at every pixel of the panorama raster view (panorama_view): the pixel's ray is sampled in every frame (sample_frames: the exact sub-pixel sensor point, the frame's bilinear value of plane, the vignetting there, the masks — saturated pixels EXCLUDED, a clipped value is no observation; the masks are fit_vignetting's, each dilated by saturation_margin_px camera pixels and symmetric over the frames: a direction any frame whose domain sees it has saturated there is measured by no frame, so the score reads the fit's own observations) and, where at least two frames measure it, ratio is the median over the other frames of log(I_k/V_k) − log(I_last/V_last), last the frame the unblended render_panorama shows (the highest index). A correct V leaves only noise, misregistration and parallax; a wrong one leaves a step at every frame boundary. The score is the ratio's spread over the pixels measured twice or more: rms and the robust mad (StatsBase's mad, normalized: the 1.4826·MAD; misalignment and moving content inflate the rms), over n pixels. gains (optional, K × 3, per frame and colour plane: fit_vignetting's metadata.gains) divides each frame's exposure gain of plane out as well, log(I_k/(g_k,plane·V_k)); nothing takes the frames as one exposure.

The camera, the masks and the frames' and planes' maps (frame_map, plane_maps) are sample_frames's (camera a VisualGeometryToolkitCore BoundedCamera: its forward map and its imaged_region). The raster is sampled in chunks of chunk pixels (the samples are chunk × K × 3 values on the frames' backend: chunk to fit a shared GPU's memory). Returns ratio (rows × columns, NaN where fewer than two frames measure the pixel) on the frames' backend.

source
PanoramaStitching.vignetting_summary — Function
vignetting_summary(v, center, frame_map, frame_size; mask) -> (; V_min, V_max, max_at, max_offset_px, asymmetry)

A vignetting v (any callable v(u) of a camera pixel, e.g. a fitted Core model) summarised over a frame's pixels: every other row and column of a frame of frame_size = (H, W), each at its pixel centre in the camera's pixels through frame_map, those mask(u) keeps (the fit's domain). V_min, V_max its range there, max_at where its maximum is and max_offset_px that point's distance to center (camera pixels: the principal point, or the coverage's centre), and asymmetry the largest |V(c + r·e) / V(c − r·e) − 1| over 40 radii to 0.95 of the farthest kept point and 72 azimuths, both points kept.

source

The model-free vignetting: the pair equation

PanoramaStitching.ring_correspondences — Function
ring_correspondences(camera::BoundedCamera, frames, rotations, directions; saturated, saturation_margin_px, frame_map, rim_margin_px, fov_deg,
                     plane, field_angle, edges, tmax = last(edges), plane_maps = ntuple(_ -> frame_map, 3)) -> PairCorrespondences

The correspondences of the pair equation: the frames sampled at directions (sample_frames, its masks and keywords; plane the colour plane read), each seen direction's sensor point put on its ring of field_angle(u) (degrees; a kernel-lawful callable of the camera pixel, e.g. the equisolid angle about the vignetting's centre) among edges, cut at tmax, and every pair of frames that see a direction in two different rings one correspondence. Each frame's saturated mask is dilated by saturation_margin_px camera pixels (≈ the registration error; fit_vignetting's one dilation), read by sample_frames' rule, and a direction any frame whose domain sees it has saturated there is dropped from every frame (fit_vignetting's one symmetric test). The sampling and the per-frame-pair selection run on the frames' backend.

source
PanoramaStitching.PairCorrespondences — Type
PairCorrespondences

The correspondences of a sweep on rings of a field angle (ring_correspondences): per correspondence the frames frames[:, i] = (k, l), the rings rings[:, i] = (a, b) (a < b; frame k sees the direction in ring a), the values values[:, i] = (I_k, I_l) of the plane read, the field angles angles[:, i] = (θ_k, θ_l) (degrees), and the ring edges edges (degrees) with the field-angle cut tmax; n_frames the frames of the sweep, n_directions the directions sampled and n_dropped those the symmetric test dropped from every frame that the frames' own masks left seen in some frame. Host arrays.

source
PanoramaStitching.ring_pair_vignetting — Function
ring_pair_vignetting(pc::PairCorrespondences; offset = 0.0, frame_weights = ones(pc.n_frames)) -> FitResult

The model-free vignetting of the correspondences pc (one deterministic computation: the algorithm, no verb): per ring pair (a, b) the weighted median of ln(I_k − c) − ln(I_l − c) (c = offset, the additive level removed; each correspondence weighted by its frames' frame_weights, a bootstrap's multiplicities), then the least squares of ln V_a − ln V_b = that median over the ring pairs, each row weighted by its summed weight, with ln V₁ = 0 (the first ring the gauge, eliminated). The estimate is ln V per ring (NaN for a ring the ring pairs do not connect to the first); the termination record (Success, ClosedForm) ((InitialFailure, NoRule) when no ring pair has a correspondence or none connects a ring to the first), rms the weighted rms of the ring-pair rows, n the correspondences used, the metadata a PairVignettingMetadata.

source
PanoramaStitching.PairVignettingMetadata — Type
PairVignettingMetadata

The diagnostics of ring_pair_vignetting: the ring edges and cut, the additive level removed (offset), the ring pairs' medians pair_median[a, b] (NaN where none) and summed weights pair_weight[a, b] (a < b), the robust residual scale residual_mad (the weighted median of |ln Ik − ln Il − (ln Va − ln Vb)|), the correspondences used per ring ring_count, and the rings the ring-pair graph does not connect to the first (unidentified: NaN in the estimate).

source
PanoramaStitching.fit_additive_level — Function
fit_additive_level(pc::PairCorrespondences; bounds, tolerance, frame_weights = ones(pc.n_frames), max_iter = 100) -> FitResult

The additive level c of the sweep's frames (I = L·V + c, the plane pc read): the root in bounds = (lo, hi) (the frames' units; a correspondence with a value at or below a level is no row at it) of additive_level_slope — the level at which the pair equation's log ratio no longer depends on the scene's brightness — by bisection to a bracket of tolerance. The estimate is c; the termination record (Success, ToleranceMet) when the bracket closed within max_iter iterations ((MaxIters, IterationCap) otherwise; (Failure, NoRule) when the slope does not change sign over bounds: no root there, the estimate the end of the smaller slope); rms the robust residual scale of ring_pair_vignetting at c, n its correspondences, the metadata (; vignetting, slope): that solution at c and the slope there.

source
PanoramaStitching.additive_level_slope — Function
additive_level_slope(pc::PairCorrespondences, c; frame_weights = ones(pc.n_frames)) -> Float64

The dependence of the pair equation on the scene's brightness at the level c: the pooled within-ring-pair least-squares slope of the residual e = ln(I_k − c) − ln(I_l − c) − (v(θ_k) − v(θ_l)) on the brightness ln(I_k − c) − v(θ_k) + ln(I_l − c) − v(θ_l) (v the ring_pair_vignetting solution at c, linear in the field angle between the ring centres and held beyond: the vignetting's variation inside a ring taken out of both, so neither carries it), each ring pair centred on its own median of e and mean of the brightness, the correspondences past 5 robust deviations of their pair's median left out. Zero at the true level (the log ratio then is the vignetting's, whatever L); positive below it (a residual level lifts the dark directions' ratio toward 0) and negative above it.

source