Spline Lens API

The grid cameras of Schöps et al. (CVPR 2020): a uniform cubic B-spline of rays over the image, central (a SplineLens of a CameraModel) or non-central (NoncentralSplineModel), and the lattice they share. Split from the Cameras API page. See the Lenses guide.

The central grid as a lens

The central grid camera as an ordinary CameraModel(K, SplineLens(…)): a backward lens on the K-normalized image plane, K a gauge whose principal point is the pixel imaging ẑ. Its optional forward spline maps the equidistant chart about ẑ to the image plane. The spline lattice's shared helpers (the grid size, the quasi-interpolant, the window of a block of controls, the chart a control moves in) live beside it.

VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(SplineLens, controls::AbstractMatrix{<:StaticVector{3}}, size; calib_area,
            intrinsics = nothing, present = nothing, unit_controls = false)
    -> CameraModel{<:LogicalIntrinsics, <:SplineLens}

THE constructor of a SplineLens camera (the source constructor and spline_lattice build through it): the camera of a pixel lattice given by its direction controls, as the reference implementation (Schöps et al.'s intrinsics0.yaml) and the stored grid records state one: controls ([gy, gx]; an absent node's control is never read, not even by the gauge) laid over the calibrated pixel area calib_area (a Rect of whole pixels, the closed rectangle of pixel corners [min, max + 1]; default Rect(size), the whole image) of an image of size, the nodes present held (nothing: every node). Its K is intrinsics (skew 0, positive focal lengths, else an ArgumentError) or, without one, the lattice's gauge. Its backward area is K⁻¹ of the calibrated rectangle, so each pixel's lattice coordinate is the reference's and the camera backprojects as the pixel grid does, whatever the K. The controls are kept as given (unit_controls = true normalizes each one unless it is unit to 1e-12, the reference reader's rule).

source
VisualGeometryToolkitCore.Cameras.CameraModel — Method
CameraModel(SplineLens, source, size; grid_size, calib_area, init = :interpolate, sampling_margin = 0.97)
    -> CameraModel{<:LogicalIntrinsics, <:SplineLens}

The SplineLens camera initialized from source — a CameraModel (its backprojection) or a function pixel -> direction (nothing or a non-finite vector where it images nothing) — over an image of size (a Size2): a lattice of grid_size = (columns, rows) controls (default default_spline_grid) over the calibrated pixel area calib_area (a Rect of whole pixels, the closed rectangle its pixels cover; default Rect(size), the whole image).

init = :interpolate (default) solves the cubic-prefilter tridiagonal system per axis so the spline INTERPOLATES the source's backprojection at the grid nodes, sampling a camera inside sampling_margin times its lens's valid domain. init = :quasi_interpolate is the 4th-order local quasi-interpolant on a lattice extended one node past the area (quasi_interpolant), with its border rule for a source whose domain ends near or inside the area; a function source takes it. The prefilter's controls are re-normalized (each unit unless it is unit to 1e-12), the quasi-interpolant's kept as it forms them.

A camera source keeps its own K (skew ≠ 0 is an ArgumentError); any other source takes the gauge of its sampled spline: the principal point is the pixel imaging ẑ (the calibrated area's centre when ẑ is not imaged inside it) and fx, fy the inverse local scale of d_xy/d_z there, skew 0.

source
VisualGeometryToolkitCore.Cameras.ControlPatchManifold — Type
ControlPatchManifold(directions[, points]) <: Manifold

A patch of a spline lattice's controls as a manifold (a SplineLens's, a NoncentralSplineModel's): a PX × PY block of its direction controls (an SMatrix of 3-vectors, in the caller's layout: the coordinates follow its column-major order) and, for a non-central model, the matching block of its point controls — the block a per-tile refinement moves (an NLLSsolver variable through the manifold adapter). A direction control d moves on its own sphere by the reference's chart,

‖d‖ · normalize(d + δ₁t₁ + δ₂t₂),   (t₁, t₂) = direction_tangents(d / ‖d‖),

2 coordinates that keep its norm (the norm is a weight of the spline's blend, not a gauge) and whose derivative at δ = 0 is (t₁, t₂). A point control moves additively, p + (Δx, Δy, Δz). The coordinates are per control, column-major over the block: (δ₁, δ₂) each, (δ₁, δ₂, Δx, Δy, Δz) with points — restrict them by mask to hold some fixed (a pupil-plane model's Δz). model(ch) is (; directions, points) (points is nothing for a central patch); retract_jacobian(ch) is block diagonal, (t₁ t₂) per direction and the identity per point.

source
VisualGeometryToolkitCore.Cameras.SplineLens — Type
SplineLens(backward::ChartSpline{3}; forward = nothing, present = nothing) <: AbstractLensModel

The central grid camera's lens (Schöps et al., CVPR 2020) as a backward lens of a CameraModel: the normalized image point x = K⁻¹u maps to the unit direction normalize(backward(x)), backward a uniform cubic B-spline of 3-vectors over the closed rectangle backward.area of the normalized plane (ChartSpline; controls [row = y, column = x], size − 3 cells per axis). backproject(lens, x) is NaN outside the area's closed box [origin, origin + widths] and where the point's 4 × 4 window holds an absent node; it never throws.

forward (a ChartSpline{2} from the equidistant chart about ẑ to x, or nothing) is the optional fitted forward map. present says which nodes the lens holds: nothing (every node) or a Bool matrix of size(backward.controls) (true = present); an all-true mask is stored as nothing, an all-false one is an ArgumentError. An absent node's control is never read.

A camera of this lens has skew 0 and positive focal lengths (an ArgumentError wherever one is built otherwise), and its K is a gauge: the principal point is the pixel imaging ẑ (CameraModel(SplineLens, source, size)). Where ẑ is not imaged inside the calibrated area, the gauge falls back to the area's centre, a pixel that does not image ẑ: the methods that read the principal point as the optical axis — undistort and undistort! onto the gauge pinhole, distortion_map, valid_rect, undistorted_rect, angle_of_view, horizon_radius and monotone_radius, image_circle(camera, sensor, d), view_overlap — then throw an ArgumentError naming the off-axis gauge.

source
VisualGeometryToolkitCore.Cameras.axis_gauged — Method
axis_gauged(camera::CameraModel{<:AbstractIntrinsics, <:SplineLens}, rotation::Rotation{3}, K′) -> CameraModel
axis_gauged(camera::Union{PosedCamera, BoundedCamera}, rotation, K′)
axis_gauged(extrinsics::Union{EuclideanMap{3}, RotationMap{3}}, rotation::Rotation{3})

The same SplineLens camera in its camera frame turned by rotation (d′ = R·d), its K the skew-0 K′ (a 3 × 3 matrix or a CameraCalibrationMat). With R taking the lens's optical axis onto ẑ and K′ the radially symmetric camera fitted to it (its principal point the pixel imaging that axis, its focal lengths and aspect the fit's), the principal point images ẑ and ẑ is the optical axis.

It is an exact re-gauge, not a fit.

  • Every backward control is turned, c ↦ R·c (with_directions), so each pixel's ray is R times its old ray, to rounding (the normalization commutes with R).
  • K becomes K′: its principal point and its focal lengths, so its aspect, are K′'s. This is CameraModel(camera, K′): any skew-0 K is an exact gauge of the lattice, so no ray changes.
  • The forward spline is dropped, since it inverts the old rays.
  • The presence mask is kept.

The poses turn with the frame.

  • A world → camera map (extrinsics, X_c = R_p·X + t) becomes (R·R_p, R·t), its frame tags kept (the third form). Every projection is then kept.
  • A posed camera turns its extrinsics with its model.
  • A bounded camera keeps its image and lens coverage, and turns its camera as above: a posed one's model and extrinsics together.

A spline that is never gauged this way keeps its source's ẑ: one read from a record, or one built by CameraModel(SplineLens, source, size).

source
VisualGeometryToolkitCore.Cameras.cubic_bspline_prefilter_taps — Method
cubic_bspline_prefilter_taps(K::Integer) -> Vector{Float64}

The one-sided taps h[0], …, h[K] of the cubic B-spline interpolation prefilter on an unbounded uniform grid: the inverse of the sampled B-spline (z + 4 + z⁻¹)/6 is √3·z₁^|k| with z₁ = √3 − 2 (Unser, Aldroubi & Eden 1991), truncated at |k| ≤ K and renormalised so h[0] + 2Σₖ h[k] = 1 (a constant is reproduced exactly). The controls c[j] = Σₖ h[|k|]·f[j + k] make the spline INTERPOLATE the samples f away from any border: the prefilter of _prefilter_matrix, the tridiagonal system of a finite lattice with its boundary rows, as a local filter a kernel can apply in a window (the dropped tail is 2√3·|z₁|^(K+1)/(1 − |z₁|) of a unit sample: 2.6e-6 at K = 10). VisualGeometryToolkitFeatures' continuous DoG response reads it.

source
VisualGeometryToolkitCore.Cameras.default_spline_grid — Method
default_spline_grid(width, height) -> (gw, gh)

The control grid the reference lays over a width × height pixel area: one node every 64 px plus the three border nodes the cubic window needs (ceil(width / 64) + 3, likewise in y; at least 4). Apply it to the CALIBRATED AREA, not the frame, when the two differ — a circular fisheye's lit field is smaller than its sensor.

source
VisualGeometryToolkitCore.Cameras.direction_tangents — Method
direction_tangents(d) -> (t1, t2)

The reference's orthonormal tangent basis of the unit direction d — the 2-dof chart a grid control moves in (normalize(d + δ₁t1 + δ₂t2)), and the plane a direction residual is measured in. Generic in the element type: the basis is in d's type (a Float32 direction gives Float32 tangents, a ForwardDiff.Dual direction dual tangents), so a device kernel or a differentiated residual is never promoted.

source
VisualGeometryToolkitCore.Cameras.forward_chart — Method
forward_chart(SplineLens) -> chart

The chart of every SplineLens's forward spline: the equidistant chart about ẑ (equidistant_chart(ẑ)), fixed in the camera frame, so no K moves it (a re-gauge maps the forward's output by the normalized plane's affinity and keeps every projection exactly). THE one derivation: project evaluates a forward on it and a fit of a forward is made on it.

source
VisualGeometryToolkitCore.Cameras.has_spline_gauge — Method
has_spline_gauge(controls::AbstractMatrix{<:StaticVector{3}}, size; calib_area, present = nothing,
                 unit_controls = false) -> Bool

Whether the pixel lattice of CameraModel(SplineLens, controls, size; calib_area, present, unit_controls) gives its own gauge K: whether its 4 × 4 window at the gauge's pixel (the pixel imaging ẑ, else the calibrated area's centre) holds only present nodes. Where it does not, that constructor refuses to build without a supplied intrinsics; a reader that can supply one asks this first. The lattice is the constructor's own (_spline_lattice: the controls normalized as it does, NaN in the absent slots), so the two agree whatever the absent slots hold.

source
VisualGeometryToolkitCore.Cameras.quasi_interpolant — Method
quasi_interpolant(samples::AbstractMatrix) -> controls

The separable 4th-order quasi-interpolant of uniform cubic B-splines: node samples f (a matrix of values with + and scalar *, laid out [gy, gx] as a spline lattice's controls) become controls c_j = (−f_{j−1} + 8 f_j − f_{j+1}) / 6, along x (the second index) first, then along y; at the border the outermost sample is replicated. Exact for cubics at interior nodes, local (each control reads three samples per axis), no linear solve. The stencil of init = :quasi_interpolate, where the samples extend one node past the grid on every side so that no grid control reads a replicated sample.

source
VisualGeometryToolkitCore.Cameras.spline_lattice — Method
spline_lattice(SplineLens, source::CameraModel, size; grid_size, calib_area) -> CameraModel

The layout a SplineLens camera built from the camera source over an image of size (a Size2) takes — the same grid_size, calib_area and K (the source's own), with the same defaults, as CameraModel(SplineLens, source, size) — WITHOUT sampling the source: every control is the optical axis (0, 0, 1). What a fit that solves for every control needs (the lattice, the gauge), with none of the node interpolation. A function source has no K without its sampled spline: build the camera.

source
VisualGeometryToolkitCore.Cameras.valid_domain — Method
valid_domain(camera::CameraModel{<:AbstractIntrinsics, <:SplineLens}) -> Rect2{Float64}

Where a SplineLens camera is defined, in pixels: its backward area mapped through K (the calibrated pixel area [min, max + 1]) when every node is present; under a presence mask, the bounding rectangle of the lattice cells whose 4 × 4 window holds present nodes only — where backproject can be finite. An ArgumentError when no window is fully present.

source
VisualGeometryToolkitCore.Cameras.window — Method
window(camera::CameraModel{<:AbstractIntrinsics, <:SplineLens}, rows, cols; directions = nothing) -> CameraModel

The block rows × cols (unit ranges into the lens's controls [gy, gx], at least 4 × 4) of a SplineLens camera as a SplineLens camera of its own: the same K, the block of the backward spline's controls (or directions, an equal-size matrix of 3-vectors, kept as given) over the rectangle of the normalized plane they span, and the block of the presence mask; no forward spline. It images a point exactly when the point's 4 × 4 window lies inside the block (and is present), and there it evaluates as the camera holding those controls, to the rounding of the lattice coordinate. It is what a per-tile cost block of a refinement evaluates: a few hundred controls instead of the whole lattice.

source
VisualGeometryToolkitCore.Cameras.with_directions — Method
with_directions(camera::CameraModel{<:AbstractIntrinsics, <:SplineLens}, D::AbstractMatrix{<:StaticVector{3}}) -> CameraModel

The SplineLens camera with the backward direction controls D ([gy, gx], kept exactly as given, in the controls' element type): the same K, lattice and presence mask. The forward spline, which inverts the old directions, is dropped (project seeds without it).

source
VisualGeometryToolkitCore.Cameras.with_present — Method
with_present(camera::CameraModel{<:AbstractIntrinsics, <:SplineLens}, present::AbstractMatrix{Bool}) -> CameraModel

The SplineLens camera holding the nodes present ([gy, gx], true = present): the same K, lattice and forward map, the controls of the present nodes kept and NaN in the absent ones.

source
VisualGeometryToolkitCore.backproject — Method
backproject(camera::CameraModel{<:AbstractIntrinsics, <:SplineLens}, pixels::AbstractVector{<:StaticVector{2}};
            device = default_device()) -> rays

Batched backprojection on the KernelAbstractions backend device (default_device() by default): the scalar backproject of the lens at K⁻¹u per pixel, one kernel (the pixels uploaded once, the lens's device copy too, the lattice slopes formed once), the rays back in the pixels' array kind — bit for bit the scalar value on the CPU.

source
VisualGeometryToolkitCore.project — Method
project(camera::CameraModel{<:AbstractIntrinsics, <:SplineLens}, rays::AbstractVector{<:StaticVector{3}},
        p₀s::AbstractVector; iters = 10, tol = 1.0e-6, device = default_device()) -> (; px, ok)

Batched projection on the KernelAbstractions backend device (default_device() by default: an accelerator when one is functional, else KA.CPU()): the scalar project's solve per ray, warm-started at the pixel p₀s[i] (a non-finite seed: the scalar project's own start) and given at most iters iterations (a lens with a forward spline reads no seed). The rays and seeds are uploaded once (a device copy of the lens too), and both outputs come back in the rays' array kind (an Array gives Arrays, a device array stays on its device with no copy). Returns the pixels and the mask of the imaged ones (converged below tol onto a present window); a pixel not imaged is NaN.

source
VisualGeometryToolkitCore.project — Method
project(lens::SplineLens, X::StaticVector{3}) -> SVector{2}

The normalized image point of the ray X (K maps it to the pixel), NaN where the lens does not image it. With a forward spline (lens.forward), its value at the ray's point of the equidistant chart about ẑ (fixed in the camera frame, so no choice of K moves it), NaN outside the forward's fitted chart (more than one knot cell past its area) or where the backward spline does not image the point. Without one, Levenberg–Marquardt on the backward spline's direction mismatch (100 iterations, converged below a 1e-6 residual) from the backward area's centre; NaN unless it converges onto a point whose window is present.

source

Exact lattice operations

A SplineLens lattice extended outward on its own spacing (the extrapolated nodes marked), and a uniform cubic lattice subdivided by integers, the same function. A resample of one spline lens onto another lattice composes them. Derivations in the Lenses guide.

VisualGeometryToolkitCore.Cameras.extend_lattice — Method
extend_lattice(camera::CameraModel{<:AbstractIntrinsics, <:SplineLens}, region; fold_spacing = 8.0)
    -> (; camera, extrapolated::BitMatrix, offset::NTuple{2, Int}, folds)

The SplineLens camera's lattice extended outward on its own spacing, with the same origin and phase, until every node whose support meets region exists. region is a region in pixels of the camera's pixel frame: a Rect (the calibrated area of the lattice a resample targets), an Ellipse, a Circle, a PolygonRegion (a decentred source's valid domain) or an intersection of them (a sensor ∩ its lens coverage). Its bounding box sets the nodes. A node's support here is open: the open rectangle node ± 2·spacing, where its basis is nonzero. So a region equal to the source's calibrated area adds nothing. (A node's support for node_mask is the closed footprint, so a node touching the region only at its edge is free there but not needed here: its basis is zero on the region.) The lattice never shrinks. The extended camera has the same K, and its backward area grows by the whole cells added, so the region lies in its valid_domain.

  • Inside the source's domain every present source node keeps its control bit for bit, so the spline is unchanged there, to the rounding of the lattice coordinate.
  • The extrapolated nodes are the new nodes, and every absent source node whose support meets the region. Their controls minimise the discrete second-order bending energy Σ Δxx² + 2Δxy² + Δyy² of the extended lattice, each difference divided by its cells' widths (difference_rows(·, 2) and its mixed rows, per component), with the present source nodes held fixed. An absent node that stays absent is not part of it. The energy is rotation-invariant in pixels (the thin-plate discretization), so the curvature, normal, tangential and mixed, does not grow outward, corners included. Its null space is the affine functions of the node indices, so an affine lattice is continued exactly; without tangential variation it is the natural rule c₀ = 2c₁ − c₂ (natural_extrapolation). Across the source's domain edge the spline is C², since a uniform cubic B-spline is C² at every knot. This penalty gives only the start values of the extrapolated nodes. It is not part of any refinement's objective: a later fit moves those nodes by its own terms. (A refinement's own rule for the nodes outside an imaged region is BoundaryRule, ring by ring from the interior.)
  • Presence. Every node is present except an absent source node whose support does not meet the region (it stays absent, NaN in its slot). A source with every node present gives a camera with every node present.
  • extrapolated marks the extrapolated nodes, a BitMatrix of the extended lattice ([gy, gx], like its controls), false for a source node kept as it was. subdivide_lattice's keyword carries it to a subdivided lattice.
  • offset is the 0-based (row, col) of the source's first control in the extended grid (window_offset's convention).
  • The forward spline is dropped, since the backward changed outside the domain. When nothing is extrapolated, camera is the input camera, its forward kept.
  • folds reports where the extended camera's backward map folds over region: folded_pixels(camera, region; spacing = fold_spacing), its folded samples, samples and unimaged, the orientation's sign taken in the source's own domain. fold_spacing = nothing skips the check (folds = nothing). The extension is C² and its curvature does not grow, but it is not guaranteed injective. A source whose rim is already folded or strongly curved continues that into the band. extend_lattice never refuses on a fold: the caller reads folds and decides (a fit can record it in its metadata). An empty folds.folded with folds.unimaged == 0 means no fold at that density.

An ArgumentError for a region whose bounding box is not a finite rectangle, or when no complete 4 × 4 window of the source is present (its present nodes then need not determine the extrapolation). Host-side lattice algebra: the result evaluates on a device like any SplineLens camera.

source
VisualGeometryToolkitCore.Cameras.folded_pixels — Method
folded_pixels(camera::CameraModel{<:AbstractIntrinsics, <:SplineLens}, region; spacing = 8.0, reference = nothing)
    -> (; folded, samples, unimaged)
folded_pixels(camera::Union{PosedCamera, BoundedCamera}, region; spacing, reference)

Where the backward map pixel → direction of the SplineLenscamera folds over region (a Rect, an Ellipse, a Circle, a PolygonRegion or an intersection of them, in pixels of the camera's pixel frame). The check samples the pixels of region on a grid of spacing px, aligned to the corner of its bounding box. At each sample it forms the orientation d · (∂d/∂u × ∂d/∂v) (backproject_with_jacobian), the signed area the map's Jacobian spans on the sphere. Being chart-free, it holds for rays past 90°. A sample folds when its orientation does not have the sign of the camera's at reference, or is zero. reference is a pixel: by default the centre of the camera's valid_domain, or the centre of its first complete 4 × 4 window where that centre is not imaged. extend_lattice gives it the source's, so a band never sets the sign.

Returns the folded samples, folded::Vector{SVector{2, Float64}}, the number of samples in region (samples), and the number of those the camera does not image (unimaged, NaN directions: no orientation, not counted as folded). The map is locally injective at that density over the imaged samples when folded is empty, and over the whole region when unimaged is zero too. A fold narrower than spacing can escape it, and an empty region has no samples. An ArgumentError when the camera does not image reference (or has no complete window), when spacing is not positive, or when the region's bounding box is not finite.

source
VisualGeometryToolkitCore.Cameras.subdivide_lattice — Method
subdivide_lattice(camera::CameraModel{<:AbstractIntrinsics, <:SplineLens}, factors; extrapolated = nothing)
    -> (; camera, extrapolated::BitMatrix)

The same SplineLens camera on a lattice whose every cell is split into m_x × m_y equal cells: its backward spline subdivided by factors = (m_x, m_y) (integers ≥ 1, or one integer for both; subdivide_lattice(::ChartSpline, factors)). The backward spline is exact knot insertion, so it is the same function everywhere. K, the backward area and the forward spline are kept (the backward has not changed). lattice_spacing is divided by factors.

  • Presence. A fine node is present when every coarse node it reads is present. The camera then images exactly the pixels the source images, and its valid_domain is the source's. An absent node holds NaN.
  • Marks.extrapolated (a Bool matrix of the source lattice's size, [gy, gx], such as extend_lattice's) marks coarse nodes. The returned extrapolated marks every fine node that reads a marked coarse node (all false without marks). No fine node that reads an unmarked node only is marked.

Host-side lattice algebra: the result evaluates on a device like any SplineLens camera.

source
VisualGeometryToolkitCore.Cameras.subdivide_lattice — Method
subdivide_lattice(spline::ChartSpline, factors) -> ChartSpline

The same spline on a lattice whose every cell is split into m_x × m_y equal cells: factors = (m_x, m_y), integers ≥ 1 (along the chart's first and second coordinate, the grid_size order), or one integer for both. The area is the same, and each axis of n controls becomes m (n − 3) + 3. It is exact knot insertion (Boehm's, every span at once): the fine controls are R_y C R_xᵀ, with R the m-ary refinement of the cubic B-spline, m⁻³ (1 + z + … + z^{m−1})⁴. The value is the same at every point, the continuation past the area included, to the rounding of the sums. Host-side.

source

The non-central grid on the lens's directions

NoncentralSplineModel: the central grid's directions (a SplineLens's backward spline) and a point spline over the same rectangle of the normalized plane, one observation line per pixel.

VisualGeometryToolkitCore.Cameras.NoncentralSplineModel — Type
NoncentralSplineModel(K::LogicalIntrinsics, directions::ChartSpline{3}, points::ChartSpline{3};
                      present = nothing) <: AbstractCamera
NoncentralSplineModel(source, size; grid_size, calib_area, init, sampling_margin)

The non-central grid camera (Schöps et al., CVPR 2020): the pixel u observes the LINE through points(x) along normalize(directions(x)), x = K⁻¹u the normalized image point, both splines laid over the same rectangle of the normalized plane with the same lattice (ChartSpline; controls [row = y, column = x]). backproject returns a Ray3; project is Levenberg–Marquardt on the reference's tangent-plane point-to-line residual. With an all-zero point spline the model is exactly its central_model, a SplineLens camera.

K is the gauge of a SplineLens camera (skew 0 and positive focal lengths, else an ArgumentError). present says which nodes the model holds, as a SplineLens's (nothing, or a Bool matrix of the grid's size; an all-true mask is nothing).

The points are plain metres in the camera frame. This is an exception to the units layering that the type forces: its controls are interpolated in the inner loop and cannot hold quantities, so the unit is stated here instead. A reader or writer of the model converts at its boundary. A PosedCamera of the model therefore takes physical extrinsics (EuclideanMap(R, t .* m), or with_unit(E, mm) of a map in mm) and projects scene points that are lengths; a plain extrinsics is an ArgumentError. ustrip(m, camera) is the plain camera, in metres only.

The model carries no image extent: a stored or rig camera holds it in its BoundedCamera.

From a source (a camera or a function pixel -> direction): the directions of CameraModel(SplineLens, source, size; …) (the same keywords and K), the points all zero.

source
VisualGeometryToolkitCore.Cameras.Ray3 — Type
Ray3(origin, direction)

A 3D ray: origin + t·direction, direction unit by contract. The unprojection currency of NoncentralSplineModel — a non-central pixel observes a LINE in the camera frame, not a direction through a common center. Its numbers are any Real: Float64, Float32 in a kernel, a ForwardDiff Dual through a back-projection.

source
VisualGeometryToolkitCore.Cameras.central_model — Method
central_model(m::NoncentralSplineModel) -> CameraModel{<:LogicalIntrinsics, <:SplineLens}

The central camera of a non-central one's directions — what a point at infinity sees: the same K, direction spline and present nodes as a SplineLens camera; the line points dropped.

source
VisualGeometryToolkitCore.Cameras.window — Method
window(m::NoncentralSplineModel, rows, cols; directions = nothing, points = nothing) -> NoncentralSplineModel

The block rows × cols (unit ranges into the controls [gy, gx], at least 4 × 4) of the model as a model of its own, as window of a SplineLens camera: the same K, the block of both splines' controls (or directions, points, equal-size matrices of 3-vectors, kept as given) over the rectangle they span, and the block of the presence mask.

source
VisualGeometryToolkitCore.Cameras.with_points — Method
with_points(m::NoncentralSplineModel, P::AbstractMatrix{<:StaticVector{3}}) -> NoncentralSplineModel

The model m with the line-point controls P ([gy, gx], plain metres in the camera frame, kept exactly as given), every other part kept.

source
VisualGeometryToolkitCore.Cameras.with_present — Method
with_present(m::NoncentralSplineModel, present::AbstractMatrix{Bool}) -> NoncentralSplineModel

The model m holding the nodes present ([gy, gx], true = present): the same K and lattice, the controls of the present nodes kept and NaN in the absent ones, in both splines.

source
VisualGeometryToolkitCore.backproject — Method
backproject(m::NoncentralSplineModel, u::StaticVector{2}) -> Ray3{T}

The observation LINE of the pixel u: the direction of the central_model (normalized) and the point spline's value (plain metres in the camera frame). A ray of NaN origin and direction where the direction lens does not image u (outside the area, or a window holding an absent node): the lens contract's form, never nothing.

T is the promoted number type of the pixel and the model: Float64 for a Float64 model, Float32 for convert_eltype(Float32, m) at a Float32 pixel, a ForwardDiff Dual at a Dual pixel. It allocates nothing, never throws and has no branch of another type, so a KernelAbstractions kernel runs it on the model's device copy (Adapt.adapt(backend, m)) on every backend.

source
VisualGeometryToolkitCore.project — Method
project(m::NoncentralSplineModel, X) -> SVector{2}
project(m::NoncentralSplineModel, X, p₀) -> SVector{2}

Project the camera-frame point X — plain metres, the point spline's unit — to a pixel: Levenberg–Marquardt in the normalized plane on the reference's tangent-plane point-to-line residual, so a converged pixel's observation line goes through X. The two-argument form starts at the area's centre (a central SplineLens's cold start); the three-argument form at the seed pixel p₀ (a good seed matters for a non-central model; positional, per-call data). NaN when the solve does not converge onto a point the model images (inside the area, its window present).

source

The lattice: a spline on a chart

The lattice of both, as a primitive: ChartSpline, a uniform cubic B-spline over a rectangle of a 2-D chart, its window and span weights, and the equidistant direction chart a forward spline is laid on.

VisualGeometryToolkitCore.Cameras.ChartSpline — Type
ChartSpline(area::Rect2, controls::AbstractMatrix{<:StaticVector{K}})

A tensor-product uniform cubic B-spline from a 2-D chart to ℝᴷ: the controls laid over the closed rectangle area of the chart (controls[row, column], the row along the chart's second coordinate, size − 3 cells per axis, so the knot spacing along axis i is widths(area)[i] / (nᵢ − 3)). Evaluated with spline_value, continued past area by the outermost cell's cubic pieces.

A CoordinateTransformations.Transformation: cs(c) is the value at the chart point c, transform_deriv(cs, c) its K × 2 Jacobian (the differentiated cubic pieces, no automatic differentiation), and cs(points) on a vector of chart points evaluates the batch with a KernelAbstractions kernel on the points' own backend (the controls are uploaded beside them).

source
VisualGeometryToolkitCore.Cameras.cubic_bspline_weight_derivatives — Method
cubic_bspline_weight_derivatives(u) -> (NTuple{4}, NTuple{4})

The first and second derivatives in u of the four cubic_bspline_weights at the span's local coordinate u: ((−(1 − u)², 3u² − 4u, −3u² + 2u + 1, u²) / 2, (1 − u, 3u − 2, −3u + 1, u)), the same cubic pieces differentiated once and twice, in the same local coordinate and Horner form as the weights. Kernel-lawful, in u's own type (a Float32 coordinate gives Float32 derivatives). The spline lattices' partials (spline_weights, spline_window_sums) and VisualGeometryToolkitFeatures' continuous DoG response read it.

source
VisualGeometryToolkitCore.Cameras.cubic_bspline_weights — Method
cubic_bspline_weights(u) -> NTuple{4}

The four weights of a uniform cubic B-spline span at its local coordinate u ∈ [0, 1): the value on the span is w₁c₁ + w₂c₂ + w₃c₃ + w₄c₄ over its four consecutive controls,

((1 − u)³, 3u³ − 6u² + 4, −3u³ + 3u² + 3u + 1, u³) / 6,

a partition of unity. Outside [0, 1) the same cubics continue (extrapolate). Kernel-lawful: in u's own type (a Float32 coordinate gives Float32 weights, a ForwardDiff.Dual dual ones), no allocation and no branch, so a device kernel can call it. Evaluated in Horner form in u, which keeps Float32 within 1e-7 of the exact weights. The weights a spline lattice's windows (spline_window) and ChartSpline read.

source
VisualGeometryToolkitCore.Cameras.equidistant_chart — Method
equidistant_chart(axis::StaticVector{3}) -> Lens(Equidistant()) ∘ LinearMap(R)

The equidistant direction chart about axis (any nonzero 3-vector, normalized here): a ray at angle θ from the axis lands at radius θ (radians) in the chart plane, in the direction of its azimuth; singular only at −axis. R is the rotation whose rows are (e₁, e₂, axis), with e₁ = normalize(ŷ × axis) (x̂ × axis when the axis is within 1e-6 of ±ŷ) and e₂ = axis × e₁ — right-handed, so the chart is the camera's own equidistant lens when axis = ẑ. The chart a SplineLens's forward spline is laid on (about ẑ), and the one a direction field over a cap of the sphere (a panorama's vignetting) reads.

source
VisualGeometryToolkitCore.Cameras.spline_value — Method
spline_value(controls, area::Rect2, point)

The uniform cubic B-spline surface of controls (any matrix of values with + and scalar *: scalars, 3-vectors) laid over the closed rectangle area ([row = y, column = x], cells = size − 3 per axis), at point, continued past the area as spline_window: a ChartSpline's value, for a lattice of any control type (a per-position aperture radius, a field over the image).

source
VisualGeometryToolkitCore.Cameras.spline_weights — Method
spline_weights(cs::ChartSpline, x) -> (ix, iy, w, (∂w₁, ∂w₂))

The weights of the 16 controls a point x of the chart reads, and their partials: w[r, c] multiplies cs.controls[iy - 3 + r, ix - 3 + c] (r the second coordinate's tap, c the first's; sum(w .* window controls) is cs(x)), and ∂wᵢ = ∂w/∂xᵢ, all SMatrix{4, 4}. The window is the point's own (continued past the area, as spline_window). In the point's precision. For a SplineLens camera's backward spline, x is the normalized point K⁻¹u: the pixel partials are ∂w₁ / fx, ∂w₂ / fy (skew 0).

source
VisualGeometryToolkitCore.Cameras.spline_window — Method
spline_window(area::Rect2, grid_size, point) -> (ix, iy, wx, wy)

The window point reads on the uniform cubic lattice of grid_size = (rows, cols) controls laid over the closed rectangle area (cells = size − 3 per axis): the controls [iy - 3 + r, ix - 3 + c], r, c ∈ 1:4, and the four weights per axis (wx[c] along x, wy[r] along y; w[r, c] = wy[r] * wx[c]). Outside the area the outermost cell's cubic pieces continue — the same continuation spline_value evaluates.

source
VisualGeometryToolkitCore.Cameras.spline_window_sums — Method
spline_window_sums(control, zero, wx, wy, dwx, dwy[, ddwx, ddwy]) -> (S, S₁, S₂[, S₁₁, S₁₂, S₂₂])

THE 4 × 4 tensor-product sweep of a uniform cubic B-spline window: the value S = Σᵣ Σ_c wy[r]·wx[c]·control(r, c) and its partials in the window's two lattice coordinates (S₁ through dwx, the first coordinate, along c; S₂ through dwy, along r), and with the second-derivative weights ddwx, ddwy the second partials S₁₁, S₁₂, S₂₂. control(r, c) returns the window's control at row tap r and column tap c (1…4), zero the accumulators' zero (a number or a static vector); the weights are cubic_bspline_weights and cubic_bspline_weight_derivatives of the span's local coordinates. One sweep, rows outer and columns inner, in plain arithmetic in that one order, so every caller sums the same bits; kernel-lawful. The spline lattices' value and partials read it (the lattice → area slope applied by the caller), as does VisualGeometryToolkitFeatures' continuous DoG response.

source

Index