Conventions

This is the stack's canonical convention text, kept once, in VisualGeometryToolkitCore's conventions/ folder. VisualGeometryToolkitCore, VisualGeometryToolkitIO, VisualGeometryToolkitFeatures, VisualGeometryToolkitEval, VisualGeometryToolkitEstimation, RANSACScoringDoneRight, LocalOptimizationDoneRight, PoseLib.jl and BlobBoards all follow it; each package's CLAUDE.md points here instead of carrying a copy, and adds only how a rule lands in that package.

Every rule below exists because breaking it produced a defect that a green test suite did not catch. Where that history matters it is stated, because a convention whose reason is lost gets "simplified" away by the next cleanup.

Naming: the verb states the kind

An estimator's name states its KIND — how it computes, and so how it can fail — and the object names WHAT IS ESTIMATED. Which family a function belongs to is decided by its input.

Correspondences or a model in → decide in this order:

  1. It needs a starting guess → refine_*.
  2. It samples hypotheses and verifies them (RANSAC) → find_*.
  3. It is one deterministic algebraic step — a minimal sample, or least squares with FIXED weights → an ALGORITHM NAME, no verb (homography_dlt, kabsch, zhang_intrinsics, tls_line).
  4. Otherwise (iterative, reweighted, repeated, staged) → fit_*.
KindInputNeeds a guessDecides inliersReturnsMust notLives in
minimal solvera minimal sample: solve(solver, cs, idx)nonoevery solution (FixedModels)iterate, weigh, rejectVGC (Homography4pt, P3P, RadialCamera5pt)
one-step solver (algorithm name)an overdetermined set taken as inliers, optionally with FIXED weightsnonothe model, plus any statistic a consumer readsiterate, reweight, rejectVGC (homography_dlt, fundamental_dlt, kabsch, chordal_mean, zhang_intrinsics, tls_line)
fitan overdetermined set taken as inliersno — it seeds itselfsoft weights at most (an IRLS)one model with residual statisticssample hypothesesRVG (fit_division_camera, fit_radial_camera, fit_camera, fit_line)
findcontaminated datanoyes: hypothesize, verify, locally optimizemodel + inlier setrequire a guessRVG (find_homography, find_absolute_pose, find_radial_camera)
refinea model and its observationsyes, requiredno; continuous reweighting at mostthe polished model, same familyre-estimate from scratch; change the familyRVG (refine_robot_world_hand_eye); PoseLib.jl's refine_* bindings; BlobBoards' board bundle

The kind fixes how a call can fail: an algorithm is one deterministic computation and can only be degenerate; a fit can fail to converge; a find is randomized; a refine can end in a local minimum. That is the point of naming by kind rather than by what comes back — the caller's error handling follows the verb.

The name follows the DESIGN, not a knob: fit_radial_camera(iterations = 1) is still a fit. find is RESERVED for the sample-and-verify search. solve is ONLY the minimal-solver protocol — VGC's solve(solver, cs, idx) and RVG's solve(problem, idx), both returning every root — never an estimator's name. The RANSAC local-optimization step is a refine: refine!(ws, problem, θ, weights) starts from the current model θ (a linear problem runs its one-step algorithm under the weights and ignores θ). The noise-model estimators (VGC's fit_scale, VisualGeometryToolkitEval's fit_prior) are outside this rule.

Detection, correspondence, observation — the terminology taxonomy (owner, 2026-11-13), enforced by check_naming (R14).

  • A detection is a detector's output in one image, unassociated: Attributed{Uncertain{<:FeatureFrame}}, a PointFrame as first-class as any scaled frame (feature-frames). A keypoint IS a detection; with a descriptor it is VisualGeometryToolkitFeatures' DescribedFeature.
  • A correspondence is a hypothesised pairing (Correspondence, Correspondences, correspondences). A match comes from descriptors; an inlier is a verified correspondence.
  • An observation is a detection associated with a model entity (a target point, a landmark), in the form an estimator consumes: TargetObservations, VisualGeometryToolkitIO's observations kind (its entity target_point or landmark), and a detection's point observation center(u). A track is the observations of one entity.
  • measured is a provenance adjective and value only (SensorSource.measured, "measured": the source of a sensor-model component such as a ChannelBlack, a covariance's), never a bound name. No identifier — function, type, field, keyword, local, constant, quoted symbol, TOML key — holds the word "measurement", the old pairing abbreviation, "sighting" or "keypoint" (R14); each old public name is in RETIRED_NAMES with its replacement (R9). A published artifact's name that spells an old word is data: it is listed in FOREIGN_FORMAT_TOKENS and accepted only in string literals, quoted TOML values and data fixtures.

Iterative-estimator vocabulary — no exceptions, enforced by a quality test in every stack package.

  1. An iteration is one cycle of block_coordinate_descent! through every block. Its counters are iter, max_iter and iterations; its hook is post_iteration!.
  2. A block is one component of the iteration state the loop overwrites: the model, the residuals, the nuisance and each of its parts (scale, inlier prior, ν, α), and the weights.
  3. update_<block>! overwrites that one block in place and nothing else. Every name that begins update_ or _update_ ends in ! and writes exactly one block; a composite block's update (update_nuisance!) only calls its parts' updates. A function that writes two blocks is split.
  4. next_<block> (or _next_<block>) is pure: it returns a block's new value and mutates nothing.
  5. <fit>_step (<fit>_step! if it mutates) is the public model-block update of the fit fit_<fit>, with weights and nuisance held fixed; that fit's update_model! calls it. fit_<fit> must exist. No other identifier contains the word step, except Manopt.jl's step_solver! (one cycle of block_coordinate_descent!).
  6. <method>_increment is a local model's Δ (_newton_increment, _lm_increment).
  7. A fixed number of reweight-and-solve cycles is iterations; a repetition of a whole estimator after changing its inputs is an outer_iteration (outer_iterations for the count). A random hypothesis draw — a RANSAC or PROSAC sample-and-verify cycle, a sampler draw, a random LO inner sample — is a trial (trial, trials, max_trials, min_trials, n_trials; its cap is the trial budget), never an iteration, and a deterministic cycle is never a trial; the one exception is VisualGeometryToolkitCore's FitResult.iterations field and its iterations(result) accessor, which for a RANSAC count its trials. A test, gate or check that holds succeeds (succeeded, n_succeeded), and one that does not fails; a caller supplies an argument. No other word is used for any of these: the banned words and phrases are listed once, in VisualGeometryToolkitCore.check_naming, which every stack package's quality test runs.
  8. The hooks of block_coordinate_descent! are exactly init_state!, update_model!, update_residuals!, update_nuisance!, update_weights!, post_iteration!, stopping_criterion, stop_reason and block_coordinate_result; its steps carry Manopt.jl's names, initialize_solver!, step_solver! and stop_solver!, and its termination tests are StoppingCriterion values with Manopt's Stop… names.
  9. The words are matched as whole _-separated tokens (CamelCase split into words) of every identifier the stack binds: functions, types, fields, keywords, locals and constants, in src, ext, test, experiments, scripts and docs/src. A method added to a function the stack does not own keeps the owner's name (Base's step(r)). Retired names — listed in check_naming — may not appear anywhere, including comments, docstrings and docs, except a CHANGELOG.md, whose job is to record what each release removed (banned words are still banned there).

Width vocabulary — one name per quantity, enforced by check_naming (R10). A robust loss is a LOSS (AbstractLossShape, DeployedLoss), never a "kernel"; "kernel" means only a GPU, convolution, minimal-solver, null-space or probability-density kernel. A width is efficiency_constant (the width as a multiple of the estimated scale ŝ — not a quantile), loss_width (a loss shape's scale parameter, the residual scale where it leaves the shared quadratic regime; sq_loss_width is its square, and a scorer's threshold is its loss's loss_width) or outlier_halfwidth (the half-width of the Student-t marginal's uniform outlier component); the noise scale stays scale / s2. A single letter or abbreviation (k, a, c, τ, thr, a *_ref or *_grid short form) is never a keyword, field, type parameter, argument, TOML key, sidecar key or CLI flag for them; a paper's symbol appears once, in the paper project's symbol map. decision_width(m, boundary) is the one conversion to the native width whose inlier/outlier decision flips at a given residual (its inverse, and a scorer's current one, is decision_boundary), and is_map_inlier(posterior_weight) the one MAP rule (posterior > ½). Each scorer takes its own width and only it: a threshold scorer (RANSAC, MSAC, the loss-width families) a threshold, the Student-t + uniform scorer an outlier_halfwidth — it has no threshold, and no name, key, comment or doc calls its half-width one. R12 flags that word close to the scorer's type or its TOML tag (six tokens); the contract's decision point is decision_boundary (a threshold scorer's constant τ; the Student-t scorer's MAP crossover at the current consensus scale, re-estimated with every score).

Images or a session in → a task verb, naming the job: find_boards (per board the answer IS a RANSAC's model and inliers), track_boards, detect_*, extract_*, load_*. A pipeline composes the estimators above and does no estimation of its own; an estimate from scratch buried inside a refine (a P3P inside a polish) is a violation to move, not to name around.

Nouns are accessors, constructors and algorithm names only — centroid(frame), blobs(bundle), cutoff(bundle), PatternConfig(...), kabsch(cs) — never pipelines: a pipeline's result's natural variable name is the noun, and detections = detections(img) makes the name local inside a function (an UndefVarError) and rebinds the global at top level. An algorithm name carries no such hazard: the result of kabsch(cs) is named R, not kabsch.

Package aliases: one short name per stack package. The registered names carry the full VisualGeometryToolkit prefix (General already holds other VisualGeometry* packages). Code that qualifies a stack package's name binds its alias once per module — import VisualGeometryToolkitCore as VgtkCore — and writes VgtkCore.foo, never the long form. The aliases are exactly VgtkCore, VgtkIO, VgtkFeatures, VgtkEval and VgtkEstimation; PoseLib keeps its name. The long name stays where Julia or a reader needs the real one: Project.toml, using/import paths (using VisualGeometryToolkitCore.Pools), a package's references to itself in its own src/ and docs/make.jl, URLs, and user-facing install snippets. A package's own tests use its alias like any other code. "Vgtk" is internal shorthand only, never a registered name.

See Robust Estimation for which function to reach for.

Signatures: the model leads, then its data, then parameters

One rule: the model or shape first, then the data it acts on or is measured against, then parameters.

CallModel or shapeDataParameters
project(camera, X)cameraX
sampson_distance(F, u₁, u₂)Fu₁, u₂
signed_distance(plane, p), closest_point(plane, p)planep
is_in_front(camera, X)cameraX
undistort(model, img, rect; scale)modelimgrect, scale
boundary_point(ellipse, θ)ellipseθ
focal_length(sensor, θ)sensorθ
centroid(ellipse)ellipse

The first argument names what the function belongs to, in the math's order (P X, H x, mul!(Y, A, X)). Measuring data against a model is the same question whether the model is F or a plane, so it takes the same order.

What counts as the model:

  • A given model leads, in task verbs too. The camera handed to find_absolute_pose(camera, pairs), the guess handed to refine_relpose(pose, K1, K2, x1, x2): estimated or given, it is applied to the data.
  • A required tool leads; a policy with a default stays last. A metric, a descriptor network and gallery, the method or scorer being evaluated: pairwise_matrix(metric, d1, d2), find_boards(gallery, network, img), evaluate_sample_budget(methods, pairs, T). But match(d1, d2, strategy = MutualNN()): a defaulted strategy is a parameter. When several model arguments lead, they keep their relative order.

Mutation follows Julia's style guide exactly ("Write functions with argument ordering similar to Julia Base"; the model-first rule never overrides it):

  • A function that modifies a Julia object argument ends in ! — a workspace, buffer, state object, or a memo cache it fills (Base's get!(f, d, k)). Not mutation, per the manual and Base: advancing an RNG (a RANSAC sampler's state counts as RNG state), writing to an IO or to files (write, mkpath, rm carry no !), and a caller-supplied scratch buffer given as a keyword whose contents don't matter afterwards (sort(v; scratch)). A positional scratch buffer becomes such a keyword; an output the caller reads back is either returned or the leading argument of a ! function (score_residuals!(sws, q, scoring, residuals, n), refine!(ws, problem, θ, weights)).
  • Base's order: (1) a function argument, (2) an IO stream — an RNG takes this slot (Random: "accept an optional AbstractRNG object as first argument"), (3) the input(s) being mutated, (4) a type, (5) inputs not mutated, (6) key, (7) value, (8) everything else, (9) varargs, (10) keywords. The model-first rule above orders the inputs within (5).
  • So only a function argument, an IO or an RNG may precede what a ! call mutates; a strategy or method object may not: map!(f, dst, src), rand!(rng, A), draw_sample!(rng, indices, problem), undistort!(dst, model, src), block_coordinate_descent!(state, likelihood, method, prob, θ₀). The in-place and allocating forms agree on the order of the rest.
  • A type argument takes slot (4), positionally — precision and element types, frame tags and problem types alike, never a keyword. In a ! call it follows the mutated inputs: undistort!(dst, [T = Float64], model, src), rand!(rng, A, S). In a non-mutating call it leads, after any function or RNG: zeros(Float32, n), rand(rng, Float32, dims), image_warp(Float32, math, rect), evaluate_sample_budget(problem_type, methods, pairs), relabel(Src, Dst, m) (like convert(T, x)). An optional type is an optional leading positional, as in zeros([T = Float64], dims).
  • An RNG leads (the IO slot): rand(rng, …), tune!(rng, method, spec, n).

Three exceptions:

  • A method added to a function the package does not own keeps the owner's order: p ∈ ellipse is Base's in(item, collection).
  • A symmetric relation takes either order: intersects(a, b), iou(a, b).
  • VisualGeometryToolkitEstimation's RANSAC family leads with the problem, which carries the data, and puts the hypothesis after it: solve(problem, idx), test_model(problem, model, idx), mahalanobis_distances!(d, problem, model), locally_optimize(problem, θ₀), and wrappers that hold a problem or a per-pair context (refit_imc(lo, ctx, seed)).

(VisualGeometryToolkitCore 0.27.0 replaced seven role-based rules with this one: ACTION vs QUERY put sampson_distance(F, u) and distance(p, plane) in opposite orders, and an image-first rule reversed the lens between the batched point undistortion and undistort!.)

VisualGeometryToolkitCore's src/base/interface_functions.jl repeats the rule beside the shared stubs.

Data arguments: correspondences in, dicts for groups

  • An estimator's data is a vector of correspondences, cs::AbstractVector{<:Correspondence} (Correspondence = Pair; correspondences(sources, targets) builds the columnar StructArray), never parallel arrays (x1, x2) or index tuples.
  • Per-correspondence data (prior scores, weights, covariances) are attribute COLUMNS (withattrs(cs; weights = w)), never keywords; per-correspondence outputs come back the same way.
  • Grouped data is an AbstractDict{K, <:AbstractVector{<:Correspondence}} keyed by the identity of the group's unknown (a marker id, a (frame, board)): fit_robot_world_hand_eye(markers), fit_division_camera(radial_cameras, views). Block order is sort(collect(keys(groups))), stored on the result; results come back keyed by K, and a joint covariance is ordered [shared | per-group in key order].
  • Given per-group models lead, keyed by the same K; the key sets must match (ArgumentError).
  • Nest only an observation with its evidence: A_i => correspondences(X, u) (a rig pose => that frame's features) in refine_robot_world_hand_eye.
  • Keywords are configuration only.
  • A value carried between stages carries its per-datum attributes as Attributed(x, attrs) (Attributed(H, (; weight)), Attributed(R_ij, (; weight))): a stage reads the weight with weight_of (1.0 for a bare value) and the value with value_of, never from a parallel weights vector or keyword. Attributed is for the data, not for wrapping results.
  • A fit returns a FitResult; its extras go in its metadata. Several fits come back keyed by what was fitted (a Dict). Per-datum outputs are index-aligned in the metadata, and callers attach them to their data with withattrs.
  • Minimal-solver kernels stay positional internals (public, not exported); the public entry is solve(solver, cs, idx).

Coordinate frames and normalization: frame in = frame out

  1. A public function (exported or public) returns results in the frame and units its inputs came in — models, residuals, covariances, inlier sets, scales: pixels in, pixels out. No public signature takes or returns normalized coordinates or a normalization, and no result or metadata carries one.
  2. Normalization is private conditioning owned by ONE estimator (fit_*, find_*, refine_*, or a one-step algorithm's public form such as homography_dlt(cs)): data mapped into the normalized frame once at entry, models composed back and covariances propagated back once at exit. The strategy is chosen in one place, CorrespondenceNormalization(M, cs; center), by model class.
  3. The inner loop never normalizes — minimal kernels, *_step updates, workspace ! forms, RANSAC problem internals and their validity gates. They take normalized data plus the estimator's CorrespondenceNormalization; a frame = nothing keyword that falls back to normalizing is a bug.
  4. Public thresholds and tolerances are in the caller's units, converted inside by the normalization's scale (similarity_scale).
  5. The one exception is the benchmark pool. Its record, VisualGeometryToolkitCore's Pools.PoolPair, stores correspondences and models in NormalizedImageFrame (image centre, half diagonal: corners at unit radius) together with the per-image maps that put them there. VisualGeometryToolkitEstimation builds a record from a pixel-frame problem (PoolPair(rng, problem, raw)) and owns the record's pool-frame RANSAC problems (HomographyProblem(pair), FundMatProblem(pair), EssentialProblem(pair)); VisualGeometryToolkitEval reads and writes pools; RANSACScoringDoneRight's pool scorers score them in that frame. Every API where pool data enters or leaves names that frame (pool_frame_maps, pool_pixel_scale, pixel_correspondences); nothing converts silently.
LayerExamplesNormalizes
task verbfind_boards, track_boardsnever; forwards pixels
estimator, one-step public formfind_homography, fit_division_camera, homography_dlt(cs)once in, once out
inner loophomography_dlt!, division_camera_step, refine!(ws, problem, θ, w), test_modelnever; takes the estimator's
minimal kernelhomography_4pt, essential_5pt, solve(solver, cs, idx)never
poolPoolPair (VGC Pools), VisualGeometryToolkitEstimation's pool-frame problems, VisualGeometryToolkitEval's pool IO, RANSACScoringDoneRight's pool scorersstored in NormalizedImageFrame, declared
function homography_dlt(cs)                       # pixels in
    N = CorrespondenceNormalization(HomographyMat, cs)    # the one strategy, private
    return homography_dlt!(A, N, cs)              # inner loop: uses N, never builds one
end                                               # H back in pixels
τ_n = similarity_scale(destination_map(N)) * τ_px # a pixel tolerance, inside

Names: the strategies HartleyNormalization(; center) (centroid, or a known centre; mean distance √2) and DiagonalNormalization(size; center) (the pool frame), applied as AffineMap(strategy, points); the two-sided value CorrespondenceNormalization, whose two sides (source_map, destination_map) move a model by COMPOSITION — D ∘ M ∘ inv(S) into the normalized frame and inv(D) ∘ M ∘ S back, with S = source_map(N), D = destination_map(N), M a map or a bare model matrix (a HomographyMat comes back a HomographyMat, an F/E a FundMat; an EpipolarMap stays one) — never by a dedicated verb (the retired frame-change verbs are listed in check_naming). *Frame names frame TAGS only (ImageFrame, NormalizedImageFrame); a normalization is never called a frame. Retired: hartley_normalize, diag_normalize, rms_normalize, HartleyFrame, coord_normalization, normalization_map.

Strip where you normalize. Physical units leave a value at the same place as its frame. The estimator that owns the normalization strips its physical inputs into its private working unit once at entry, with the whole value's ustrip(u, X), and reattaches the unit once at exit, where it composes the model back. Every layer below it sees plain numbers. A task verb forwards quantities as it forwards pixels, and a minimal kernel never sees a unit. See "Units" (units.md) for the layering and its four enforcement points.

Calibrated geometry (E, relative pose, PnP) is on unit bearings, never the z = 1 plane, and takes no normalization: see "Calibrated geometry is on bearings" (bearings.md).

Each package states how the rule lands in it in its own CLAUDE.md ("In this package", "Frames here").

Why this rule is worth its cost

A tolerance is a number in a frame, and neither the type system nor a test suite carries the frame with it. One constant in this stack — the coincidence tolerance that rejects duplicate detections — was read in three frames by three consumers, none of which names the helper, so grepping its call sites finds none of them: as metres in P3P's world points (a 10 cm radius), as a spread-normalized unit (≈17 px at full frame), and in a calibrated chart (120 px at f = 1200, rejecting 22% of clean minimal samples).

None of those showed up as a failure. An over-strict gate makes RANSAC discard more samples, so the suites go quieter, never red. The question that finds this class of bug is: what units is my data in, at every problem I construct?

The frames of a rig, its members and a capture

A Rig is posed in ONE frame F (a body BodyFrame{:camera_rig}, the world, a tracker's room, or a member's own frame — the gauge). Each member is a (model, pose) pair and names its own frame:

  • a sensor's is ViewFrame{id}; its pose is its extrinsics, F → ViewFrame{id};
  • a target's is TargetFrame{id}; its pose (PosedTarget) is TargetFrame{id} → F, stored unlabeled and minted on access. The target's geometry — points(target), a PlanarTarget's plane z = 0 (or the 3-vectors a HeightField calibration lifts off it) or a PointTarget's points — is in TargetFrame{id} and independent of the pose, so one target can sit in several rigs.

A capture's per-frame values keep the same frames: a Trajectory maps one body's frame into one frame over time (BodyFrame{:board_rig} → WorldFrame), TargetPoses map TargetFrame{id} → F per frame (F the camera's frame), and TargetObservations{F} are pixels in a declared PixelFrame or points in a declared 3D frame, the dimension the frame's (one container for both since 0.10). An observation names a target point by (target id, point index) — in TargetObservations, per frame and target, index => observation — resolved against the target's definition, the rig member.

A planar target's frame: one orientation for every design

Every planar target's TargetFrame{id} has the same axes: x right and y up as the target is seen from its viewing side, and +z towards the viewer, so the design lies on z = 0 and a camera that sees it has its centre at z > 0 (signed_distance to the plane > 0). The viewing side is the printed side of a print; a target that is seen from behind (backlit, transparent, a mirror) states its viewing side, and every cheirality test reads that side rather than assuming one.

The ORIGIN is the design's own and is documented with it: an AprilGrid's is tag 0's first corner (Kalibr's frame, point for point), a SchoepsPattern's the pattern's lower-left corner, a BlobBoards board's its centre. An origin is a translation, never a rotation, so the orientation, the side and every cheirality test read the same for every design; where the printed page sits relative to the origin is the target's Canvas.

A tool's own target frame is converted once, at the IO adapter that reads its files, by a fixed rigid map, and nothing inside the stack sees it: Schöps' KnownGeometry (y down: a half turn about x and a shift of one square), and OpenCV's and MATLAB's checkerboard frames (z into the board, the camera at z < 0). Kalibr's AprilGrid frame is this one.

Why one orientation. A front end that built its own page frame (y down, from texture pixels) and searched both row directions found two labellings of one scene — rows along −y with the camera behind is rows along +y rotated 180° about x — scored them alike, and let rounding choose: Julia 1.12 and 1.13 chose differently (RadiometricCamera, 2026-11-05). With the axes fixed by the convention there is one labelling, and cheirality removes the one remaining wrong pose, the camera behind the target.

Pixel origins

Three canonical conventions, no aliases:

  • :opencv — top-left pixel centre at (0, 0)
  • :colmap — top-left pixel centre at (0.5, 0.5)
  • :julia — top-left pixel centre at (1, 1)

:vlfeat, :makie and :matlab were removed as redundant aliases; do not reintroduce them.

Geometric operations default to COLMAP: Rect(sensor) has its origin at (0, 0) with widths equal to the resolution, and the image-display functions (imshow, CampixelImshow) and the other geometric APIs follow. Conversion between conventions is an EuclideanMap(PixelFrame{A}, PixelFrame{B}) — see Coordinate Frames. Persisted files declare their convention; a reader returns typed data rather than a bare array, so the frame travels with the numbers instead of living in a comment.

Calibrated geometry is on bearings

Calibrated two-view and pose geometry — the essential matrix, relative pose, P3P/PnP, their RANSAC problems, refiners, scoring and benchmark pools — works on unit bearings (unit rays on S², typed StaticVector{3}), never on the affine z = 1 plane (GnomonicFrame, the chart).

The reason is not tidiness: a ray at or beyond 90° has no coordinate on that plane, so any K⁻¹u dehomogenized to a 2-vector, any silent [x, y, 1] lift and any scoring through F = K⁻ᵀEK⁻¹ excludes the fisheye and omnidirectional cameras this stack calibrates. Calibrated solvers and problems are typed Correspondence{<:StaticVector{3},<:StaticVector{3}} so a 2-vector is a MethodError rather than a silent narrowing.

The type is a necessary condition, not a guarantee. A three-vector still on the z = 1 plane, (x, y, 1), satisfies the signature, and because the epipolar algebra is scale-invariant per endpoint it even returns the right answer for every ray the chart can represent. What it cannot represent is a ray at or past 90°, so the fisheye exclusion survives the type and merely stops being visible in it. The guarantee comes from where bearings are built — the camera's back-projection, defined for every ray the camera can see — and the essential problem checks unit norm once, at construction. A unit-norm check inside the solver was considered and deliberately not taken: it would be a per-sample tolerance in a hot loop, checking data the caller built. At the boundary it is a structural discriminator rather than a tuned constant, since a z = 1 point has norm ≥ 1 (≈ 1.41 at 45° off axis) and coincides with a unit bearing only on the optical axis, where the two are the same vector.

Consequences worth stating:

  • Bearings come from the camera's own back-projection, backproject_with_jacobian(camera, u) -> (f, M), whose Jacobian is tangent to the sphere (fᵀM = 0).
  • The residual is a tangent-plane (spherical) Sampson distance whitened by the pixel covariance propagated through that Jacobian, which keeps thresholds in pixel σ where the caller set them.
  • Calibrated data takes no normalization: only rotations preserve the essential structure, and unit bearings need no conditioning.
  • The tangency assertion fᵀM = 0 is load-bearing, not decorative. Swapping the two views' Jacobians moves the residual by under 2% (0.19% at equal focal lengths) but its Jacobian by 341%: a swap destroys the descent direction, not the cost, and is invisible to Monte Carlo calibration averages. Only tangency, checked per view, catches it — and a test of per-view wiring needs two different focal lengths to be able to fail at all.

Three declared pinhole islands, one fact with three instances. A matrix cannot express a general camera's nonlinear back-projection, so (1) the mapped-homography path, where K⁻¹ declares the calibration as observation maps, and (2) VisualGeometryToolkitEstimation's relative-pose bundle, which holds K matrices rather than camera models, are pinhole by construction. (3) LocalOptimizationDoneRight's vendored RansacLib baseline is third-party and exempt. Making (1) or (2) camera-general means carrying camera models instead of matrices — a separate decision, not a rider on this rule. Separately, the Monte Carlo harness's epipolar band trial keeps its query in the z = 1 chart (an epipolar line is a calibrated-plane object), so it cannot validate an essential matrix's covariance at a fisheye periphery: a green band trial is not fisheye coverage.

When checking whether something "runs on bearings", trace the data PREPARATION path, not just the kernel. The five-point kernel had taken bearings all along, while the affine-plane assumption was found in five other places by five different routes: a pool generator dehomogenizing K⁻¹u, a 2-vector lift the solver registry documented as "the caller's frame convention", a cheirality test building (x/z, y/z, 1), a helper named for bearings that returned plane points, and the Monte Carlo harness's own interface — the last of which no test could catch, because the harness is what the tests are written in.

Observation noise lives on the observations

A correspondence residual is a constraint g(θ; x₁, x₂) = 0 whitened by the noise of its observations. That noise is a property of each OBSERVATION, stated where the observation is built, never a property of a problem type:

EndpointCovarianceSpelled
plain pointΣ = I, unit isotropic noise in its own framep
uncertain pointits own Σ (or precision)Uncertain(p,CovMat(Σ)), Uncertain(p,PrecMat(Λ))
exact pointΣ = 0exact(p)

With G_x = ∂g/∂(x₁, x₂) = [G₁ G₂], the constraint covariance is the first-order propagation of both endpoints and the residual is the constraint whitened by it:

Σ̃ = G₁·Σ₁·G₁ᵀ + G₂·Σ₂·G₂ᵀ,    r = L⁻¹·g,    L·Lᵀ = Σ̃.

pair_covariance is the 4×4 block diagonal over (x₁, y₁, x₂, y₂) that whiten and whitening_factor take as Σ_x. A correspondence whose two endpoints are both exact is an error (ArgumentError): Σ̃ = 0 leaves nothing to whiten, and no residual unit exists.

Two layers, two distances. The noise never enters the Sampson distance: sampson_distance(M, x…) is GEOMETRIC, the first-order distance from the observations to the model's variety in their own units, whatever covariance they carry. Its residual in the constraint's normal basis, sampson_residual_with_normal(M, x…) = (s_N, N), is what the noise layer whitens: r =whiten(s_N, N, Σ) = L_N⁻¹·s_N, L_N L_Nᵀ = NᵀΣN, which is exactly L⁻¹·g above (G_x = L_g·Nᵀ, so Σ̃ = L_g·NᵀΣN·L_gᵀ). mahalanobis_distance(M, x…) is ‖r‖ with Σ read from the observations by this rule: for plain points it equals the Sampson distance.

What a threshold means.r is in units of the noise. For plain image points that is the pixel: a 3 px threshold means 3 px, exactly as before this rule was written down. With real covariances r is in SIGMAS — r² ~ χ²(d_g) — and a threshold is a number of standard deviations. An exact source with a plain target is the plain pixel case on one side: the residual is the transfer error in the target image.

Plain is bit-for-bit what it always was. A plain set carries no covariance columns, every reader takes its Σ = I path unchanged, and the stack's equivalence goldens are identical across the change. Image↔image matches, pool pairs and F/E/H estimation on them do not move.

Why the old formulation was wrong — a worked example

Until 2026-10 every correspondence problem hard-coded Σ₁ = Σ₂ = I over (x₁, y₁, x₂, y₂), i.e. assumed both endpoints were noisy points measured in the SAME unit. A calibration board breaks that silently.

Take a 5 cm board in METRES seen at 0.3 m by an f = 800 px camera. The board → image Jacobian is about f/Z ≈ 2670 px/m (measured: singular values 2500–2600 at the board centre). For the homography's transfer form G_x = [A −I], A = ∂u₂/∂x₁, so

Σ̃ = A·Aᵀ + I ≈ 2600²·I,    r ≈ e / 2600,

with e the transfer error in pixels: the residual comes out in BOARD METRES. A 0.2 px inlier scores ≈ 7e-5 and a 40 px mismatch ≈ 0.015. Both are far below a 3 px threshold, MSAC counts everything as an inlier, and the fit lands tens of pixels off (25–40 px across draws of the regression test, VisualGeometryToolkitEstimation's metre-unit board).

With the board exact, Σ₁ = 0: Σ̃ = I, r = e in pixels, and the 40 px mismatch is rejected at 3 px. The fit is then the classical reprojection estimator for a known target.

Why it went unnoticed. Before the K+ refactor the default scorer was a scale-estimating marginal (Student-t + uniform): it estimated the residual scale from the data, so a residual in metres simply produced a scale in metres and the classification came out right. MSAC's FIXED pixel threshold, made the find_* default in K+, had no scale to absorb the unit and exposed it. A scale-estimating scorer still hides a unit error; it does not make one correct — its outlier half-width is still stated in the residual's unit.

When to use which

  • Image ↔ image matches (detections in two images, pool pairs): plain.
  • Board, model, world or map points — anything known by construction rather than measured: exact. A board → image fit is exact.(board) .=> pixels.
  • Detections with a covariance (a detector's per-point uncertainty, a propagated estimate): Uncertain, in the caller's frame.

A problem class whose observation model already treats one side as exact is consistent with the rule as it stands: absolute pose takes plain 3D points and whitens only the image point; the radial homography and radial camera score the reprojection distance in the target image, accept their board side exact (or plain, its long-standing meaning there) and refuse any other covariance. A problem class whose every residual route (scoring, refit rows, lift rows) does not yet read covariances refuses Uncertain/exact endpoints with an ArgumentError rather than scoring one function and descending another (VisualGeometryToolkitEstimation: F and E today).

Normalization

Covariances are in the CALLER's frame, like the points (frame in = frame out). An estimator's private normalization moves them with the points: an affine side map x ↦ A·x + b takes Σ to A·Σ·Aᵀ (s²·Σ for a similarity of scale s); normalize(N, cs) of a CorrespondenceNormalization does it for the correspondence set's covariance columns, and a projective observation map uses its Jacobian at each point. The Mahalanobis residual is invariant to the move, so thresholds stay in the caller's units — pixels for plain pixel data, sigmas for covariances — and nothing downstream converts them.

Storage

noise_columns turns Uncertain endpoints into two attribute columns of the correspondence set, Σ₁ (sources) and Σ₂ (targets), static 2×2 matrices in the caller's frame; a side whose endpoints are all plain has NO column (unit isotropic). An estimator's constructor calls it once; the inner loop reads the columns (pair_covariance(cs, i), the batch rows whitened_residuals! / whitened_jacobian_rows!) and never an Uncertain value. A RANSAC problem hands its point's covariance to the scorers through ONE hook, observation_covariance(problem, i) (I by default), never as a slot of its constraint tuple: constraint_with_jacobians(problem, model, i) is (g, G_x, G_θ) for every problem, as for every model. exact is an ordinary zero CovMat: its four numbers per point cost nothing past the constructor.

Feature frames: one interface, every verb once

A detection is a FeatureFrame, plain or Uncertain, in an Attributed with its metadata. The family is ordered by degrees of freedom (frame_dof): PointFrame 2 (a centre), Disc 3 (+ σ), OrientedDisc 4 (+ θ), EllipticFrame 5 (a symmetric shape factor), AffineFrame 6 (a general one). The DOF level is a trait, frame_scale: PointShape(), IsotropicScale(), ShapeFactorScale().

Every verb is implemented ONCE, on FeatureFrame or on its trait, from the per-type primitives: the centre (centroid), the local shape where a frame has one (shape_factor; σ for the isotropic frames), and two one-line constructors per type (its frame from mapped lengths, its similarity image). A PointFrame has only the centre, so it gets the whole interface through the hoisted code — no verb has a copy per type. It is isbits with the size and layout of Point2{T}, and its verbs cost what the Point2 verbs cost (tested: no allocation, no Union or boxing in the typed code, a KernelAbstractions kernel on the CPU and the GPU over a StructArray of detections, and a benchmark against Uncertain{Point2}).

The verbs every frame has

VerbEvery frame (PointFrame included)
centrecentroid(f) (also GeometryBasics.origin)
the point observation of a detectioncenter(u::Uncertain{<:FeatureFrame}), center(d::Attributed{<:Uncertain{<:FeatureFrame}}) → Uncertain{Point2}: the centre and the first 2×2 block of the frame's covariance. THE one conversion: no code takes a centre and a covariance block apart by hand
similarity actionS(f) for a SimilarityMap{2}, an EuclideanMap{2}, a rotation or uniform-scaling LinearMap: exact, the class kept
transfer through any point maptransfer(m, f): the centre by m; a scaled frame's shape by m's Jacobian at the centre (an AffineFrame, first order), exact through a similarity
propagatepropagate(m, u::Uncertain{PointFrame}) through every framed map an uncertain point propagates through (EuclideanMap, SimilarityMap, HomographyMap, FramedAffine), the frame kept; for a scaled frame, propagate(m, center(u))
error ellipseEllipse(u::Uncertain{<:FeatureFrame}; sigma): its centre's
bounding boxextrema(f), Rect2(f), Rect2{Int}(f) (the 1σ box), bbox(f; scale, snap_to_grid) (the scale-σ box, unit-stripped; a point's is the point at any scale)
unitsustrip(u, f), f * s, f / s (a change of unit or length scale; angles kept)
translationf + v, f - v
GPUisbits values: adapt is the identity, a StructArray of (uncertain, attributed) frames is columnar
codecVisualGeometryToolkitIO's one detection codec: center and the shape, covariance over the frame's parameters, centre first (frame_dof; persisted-geometry)

The verbs only the scaled frames have

VerbDisc, OrientedDiscEllipticFrame, AffineFramePointFrame
scale(f)σ— (read characteristic_scale)absent
shape_factor(f), shape_matrix(f)σI, σR(θ); σ²I exactlyS, A; F Fᵀabsent
characteristic_scale, aspect_ratioσ, 1`√det F
AffineMap(f), boundary_point(f, θ)the canonical → image mapthe sameabsent
region(f, k), Circle(f, r), Ellipse(f, r)the k·σ contourthe sameabsent
MvNormal(f)N(c, σ²I)N(c, F Fᵀ)absent
inv(f)Disc: (-c/σ, 1/σ); OrientedDisc absentthe class kept: (-F⁻¹c, F⁻¹)absent
intersects(p, q, k)Disc onlyabsentabsent

An absent verb is absent BY DESIGN, not a stub: calling it on a PointFrame is a MethodError whose hint names the interface and the point's own verbs.

Residuals and distances: the prefix is the currency, the noun is the shape

Every name in the residual and distance family states two choices, each in a fixed place.

The currency is the prefix.

  • sampson_ is GEOMETRIC: the first-order distance to the model's variety in the data's own units, whatever noise the observations carry.
  • whitened_ and mahalanobis_ are divided by the observation noise: in σ units, which for plain points is the pixel ("Observation noise lives on the observations", observation-noise.md).

The noise comes from the observations (plain, Uncertain, exact), never from a type switch on one name and never from a whitening the caller supplies. One name for two currencies is what "Derivatives and Jacobians" forbids (jacobians.md), and for the same reason: for plain points the two currencies agree, so a caller who confuses them stays green on every plain test and is wrong on the first covariance.

The shape is the noun. With d_g the constraint's codimension:

NounValueSign
residualthe vector of d_g componentssigned
distanceits norm ‖r‖≥ 0
signed distancethe residual when d_g = 1: one numbersigned

signed_distance(plane, p) is the geometric signed distance (VisualGeometryToolkitCore 0.5.7). distance(hyperplane, p) was removed in 0.7 because it returned a signed value under an unsigned name.

A square is named as one.‖r‖² is the SQUARED distance: gᵀΣ̃⁻¹g is the squared Mahalanobis distance (~ χ²(d_g) under H₀), never "the Mahalanobis distance". A threshold on a distance and a threshold on its square differ by a square root, so a square under the plain name is a threshold off by a power.

Precedent. Statistics: a residual eᵢ is signed, the Mahalanobis distance √(rᵀΣ⁻¹r) is ≥ 0 (R's mahalanobis returns the square under the plain name, the confusion this rule avoids). Ceres: residuals are signed vectors and the cost is ½‖r‖². GTSAM: whitenedError and unwhitenedError are signed vectors and error() is the scalar.

The names, by currency and shape:

CurrencyResidual (signed vector)Distance (≥ 0)Signed distance (d_g = 1)
geometricsampson_residual, sampson_residuals!sampson_distance, sampson_distances!signed_distance
whitenedwhitened_residuals! (model level (r, M, xs) and problem level (r, problem, model)), whitened_residual(problem, ctx, i) (one point)mahalanobis_distance (model level (M, x…) and problem level (problem, ctx, i)), mahalanobis_distances!(d, problem, model)the whitened residual, d_g = 1

The RANSAC protocol

The constraint-problem protocol (VisualGeometryToolkitCore's robust/problem.jl) follows the same table. What a scorer reads per point is a DISTANCE: mahalanobis_distances!(d, problem, model) fills d[i] ≥ 0 from the per-point hook mahalanobis_distance(problem, ctx, i), ctx the problem's residual_context(problem, model) (the model by default). The signed residual is whitened_residual(problem, ctx, i), in the constraint's shape (a number at d_g = 1, its d_g components otherwise), and whitened_residuals!(r, problem, model) stacks its rows. The squares are sq_distances and square_distances!. Scores read d², which the residual's sign does not move. Until the 0.7 wave the per-point scalar was a signed distance at d_g = 1 and a distance otherwise, under the word "residual"; a whitened_residual override written then returns a scalar at d_g ≥ 2, and test_constraint_problem refuses its shape.

Persisted geometry: uncertainty and storage

How geometry is written to and read from files (VisualGeometryToolkitIO's vgio/<kind>/v<major>.<minor> JSON kinds and any other stack format). It continues the observation-noise rule above: the same three cases, now on disk.

Uncertainty mirrors the type

Every geometry-carrying record, whether a primitive (a 2D or 3D point) or a pose (a rigid transform), stores uncertainty exactly as the in-memory type states it:

In memoryOn diskRead back as
plain Tno covariance fieldplain T
Uncertain{T} (Attributed{T, CovMat})covariance presentUncertain{T} with that CovMat
exact(p) (zero covariance)covariance all zerosUncertain{T} with a zero CovMat (= exact(p))
  • A writer emits covariance if and only if the value is Uncertain. The key is covariance in every kind and every body, in one of the three storages below; no kind spells it otherwise (a flat vector, a cov with a separate size).
  • An absent covariance is stated by its producer. A record whose producer has no finite covariance omits the key and states a per-record covariance_status (for example "unrefined") with its reason. A null covariance is refused, and a reader never substitutes a default for an omitted one (the producer owns its observation policy).
  • A reader returns the matching type, so a value round-trips exactly, type included.
  • A precision-form Uncertain (PrecMat) is written as its covariance and read back in covariance form.
  • Plain means "no covariance stated", not "unit noise". The consumer applies its own documented default. For an image point that default is the observation-noise rule's Σ = I; other consumers document theirs.
  • exact means zero, and it is written as zeros.
  • An estimator states its uncertainty. A file written from an estimate (a hand-eye calibration, a fitted pose) carries Uncertain values. A writer that has a covariance and drops it is a bug.

One encoding per storage

The covariance states the storage the value's CovMat holds, so it round-trips as that storage:

In memory (parent(Σ))On disk
SMatrix (full)a d×d array of rows (d arrays of d numbers), symmetric
SDiagonal (diagonal){"diagonal": [d numbers]}
ScaledIdentity (isotropic, λ·I){"isotropic": λ}

An isotropic covariance over entries of different length dimensions (a pose's angles and lengths) has no single number in the file's units, so it is written, and read back, as its diagonal. The storage is told from the JSON's shape, never from the numbers. Every form is in the primitive's own tangent order and units:

  • Points:(x, y) or (x, y, z), in the file's unit squared. That is px² in a file that declares a pixel_frame, and the declared length_unit squared otherwise.
  • Poses: 6×6 over [δω; δt], rotation first: rad² for the rotation block, the file's length_unit squared for the translation block, and their product for the cross block.
    • The chart is VisualGeometryToolkitCore's RigidExpManifold6: R ↦ exp(δω)·R and t ↦ t + δt. The rotation is perturbed on the LEFT, in the pose's destination (to) frame, and the translation additively in the same frame. This is the chart VisualGeometryToolkitEstimation reports estimator covariances in.
    • Relation to GTSAM. The order [ω; v] is GTSAM's Pose3 tangent order (Pose3::Expmap: w = xi.head<3>(), v = xi.tail<3>()).
    • GTSAM's chart is different. It is the RIGHT (body-frame) retraction T·Exp(ξ), so a GTSAM marginal needs the adjoint, not a relabel.
    • Relation to Sophus. Sophus' SE3 tangent is [υ; ω], translation first.
  • ROS PoseWithCovariance. It is row-major 6×6 over (x, y, z, rot X, rot Y, rot Z), with fixed-axis rotations in the parent frame and the translation in the parent frame. That is this chart with its blocks swapped.
    • The conversion is the permutation P = [0 I₃; I₃ 0]: Σ_ros = P·Σ·Pᵀ.
    • VisualGeometryToolkitIO's ros_pose_covariance and pose_covariance_from_ros implement it.

Units: memory in base units, files in the declared unit. In memory a covariance is in base units, like the value it belongs to: m² for a 3D point, and for a pose rad² in the rotation block, m² in the translation block and m·rad in the cross block. On disk it is in the file's declared length_unit. A writer converts each entry with VisualGeometryToolkitIO's write_length rule, the same decimal shift as the value, raised to the entry's power of length, and a reader converts back the same way. A file in mm therefore holds mm² and mm·rad entries, and a covariance read from it is in m² and m·rad again. Pixel covariances are px² and are never converted.

A reader refuses a covariance of the wrong size, a non-finite one, or an asymmetric one. It does not test positive semi-definiteness (an exact zero is valid). Every matrix a file holds (a covariance, a shape matrix, an affine shape factor) is written in rows, the stack's MATRIX_LAYOUT.

An uncertain feature frame's covariance is in the frame's parameter order, the centre first. A detection's FeatureFrame is written as its center and its shape parameters (frame_dof of them), and its covariance is over exactly those parameters, (x, y) first: a PointFrame's is the uncertain point's 2×2, so the point frame is the 2-DOF case of the one frame codec, never a codec of its own. The point observation of a detection is its first 2×2 block (center(::Uncertain{<:FeatureFrame})), read by that function only.

One units object, one conventions object

Every file states its units ONCE, in one file-level units object, and its conventions once, in one conventions object; neither has a default, and a file without either, or without one of their keys, is refused:

"units": {"length": "mm", "time": "s", "angle": "rad", "pixel": "colmap",
          "density": "dpi", "radiance": "relative", "length_roles": {…}},
"conventions": {"index_base": 1, "matrix_layout": "rows", "rotation_encoding": "quaternion_wxyz", …}
  • A field's unit is a power of a declared base, stated by the schema, never by the file. A blur covariance is length² on a print and px² on a sensor; a focus surface is length⁻¹. No block carries its own "unit" string.
  • Time values are in units.time; a time_origin is ISO 8601 with its UTC offset.
  • Angles are radians, declared as the fixed token "rad".
  • Sizes are objects. Every Size2 (an image size, a board, canvas or texture size, a pixel pitch, a lattice's grid_size, the identity tree's sizes) is {"width": w, "height": h}, never a positional pair. A rectangle (area) and the positional primitives (an ellipse, a circle) stay documented arrays, described field by field in the kind's JSON Schema.
  • Every index is 1-based, Julia's, declared once as "index_base": 1 in the conventions object: a point index, a lattice's present runs, a frame index. Only a foreign format's importer and exporter (Schöps, COLMAP, OpenCV, all 0-based) convert, at their own boundary; a stack reader and writer hold no ±1.

Versions: a major and a minor

A schema tag is vgio/<kind>/v<major>.<minor>. A reader accepts any minor of its major: a minor version adds optional keys its schema declares, and an unknown, undeclared key stays refused. A major version changes a meaning. The versions a reader accepts are derived from its migration steps, never kept as a list by hand.

Storage: JSON now, a binary payload past a threshold

Every kind is a single JSON file today. A kind keeps that form while its bulk numeric arrays stay small.

Once a kind's arrays exceed about 10 MB, it follows glTF's JSON + bin pattern:

  • The JSON header stays. It holds the schema tag, the declared conventions, the frames, the sensor and the provenance.
  • The bulk arrays move to an Apache Arrow sidecar. The header references the sidecar under the reserved top-level key "payload": {"file": "<name>.arrow", "sha256": "<hex>"}.

Every kind reserves "payload" now, so a sidecar can be added later without a schema break. No kind implements it yet. Until one does, a reader refuses a file that carries "payload".

Every stored kind is a VGC type

A persisted kind is the DIRECT serialization of a VisualGeometryToolkitCore type: the reader returns the VGC value, the writer takes it, and the round trip is exact by type (plain ↔ plain, Uncertain ↔ Uncertain, labels ↔ frame names). No format-side record type stands between the file and the value; a foreign format (COLMAP, an OpenCV or MATLAB export) keeps an adapter type that converts to and from VGC types.

Kind (VisualGeometryToolkitIO)VGC value
camerasDict{Symbol, …} of unposed BoundedCameras of a CameraModel (name → camera)
transformsa Vector of labeled EuclideanMap{3}s (Uncertain allowed)
rigRig: sensors, PosedTargets, or both
targetan AbstractTarget (PointTarget, PlanarTarget)
trajectoryTrajectory
target_posesTargetPoses
observationsTargetObservations (pixels or 3D points, by its frame; each set names its entity, target_point or landmark). The 0.9 target_detections, target_points2d and target_points3d kinds are read through version steps into it
detectiona detection, Attributed{Uncertain{<:FeatureFrame}} (the one frame codec, every frame type)
sensor_characterisationSensorCharacterisation: the body and, per ISO, one body per present component (an absent one absent), each with its source (measured, estimated, specified, file)
capture_settingsCaptureSettings, Attributed with its identities' sources

File names state the kind, never the time. A file's name is a convention for people, not a format: every loader reads the file's schema and never its name. VisualGeometryToolkitIO's docs ("Kinds and the I/O Surface", File names) state it once.

The observation kind is COLMAP's decomposition: an observation ≈ COLMAP's images.txtpoints2D with their POINT3D_ID — here the pair (target id, point index) — and the target rig ≈ points3D, kept as per-target frames (each target's points in its own frame, placed by its pose) with no stored tracks.

Metadata: envelope for the file, attributes for the value. The file's envelope holds what is about the FILE — the schema tag, the declared conventions (the units and conventions objects), the generator (the writing package and its version), the reserved payload, and the free-form provenance, which is written from a keyword and never read back — and none of it is part of the value. A fact about the VALUE that must round-trip is part of the VGC value: a field (a trajectory's per-frame images and times, an observation's target identities) or an Attributed attribute — a covariance as Uncertain, a camera's calibration block as Attributed(camera, (; calibration = (; tool, source, date, reprojection_rms, n_images, n_points, note, device))) (device = (; body, lens), the DeviceIdentitys of the hardware it was made for: values are math, identity is an attribute, so a camera model holds no names), a target's as Attributed(target, (; calibration_record = (; …))) (the same closed set of keys; the target's own calibration is its TargetCalibration, part of the value, so the record is named apart).

A target's calibration is part of its record. A PlanarTarget's TargetCalibration is written in its body as one calibration record (vgio/target/v4, vgio/rig/v7): its type, the kind's tag in TARGET_CALIBRATION_TAGS; its parameters in parameter_names order, lengths in the file's unit; nodes for a thin-plate kind; for a composition (the affine print error Aspect ∘ Skew, HeightField ∘ Aspect ∘ Skew, HeightField ∘ ThinPlateSpline, …), written by its structure, {"type": "composition", "compose": [outer, …, inner]}, the members in CoordinateTransformations' order, the outer map first; and a covariance over every parameter, in that order, iff it is Uncertain. A lens record lists its stages in sequence under the same key, compose, in the same order. A list whose order changes changes its key: the retired calibration_chain / members and lens chain listed the same members in application order, and a reader refuses them by name in a current record rather than read them in the new order (a file of the version that held them is read through that version's migration). A target with a board (the physical board it is printed on) writes it as {"size": {"width", "height"}, "origin": [u, v]}, lengths in the file's unit.

A referenced design is stored by its reference. A PlanarTarget whose design is content-addressed (a BlobBoards board uid) is written by its reference and its calibration; its nominal points are the design's owner's and are not restated. A reader returns it unresolved unless the caller supplies the owner's resolver.

A parametric design is stored by its layout. A PlanarTarget built from a layout (PlanarTarget(layout; …), an AbstractTargetLayout: AprilGrid, SchoepsPattern) records it (layout(t)) and is written by it (vgio/target/v4, vgio/rig/v7): a layout record whose type is the layout's tag in TARGET_LAYOUT_TAGS, its whole design (tag family, ids) and its lengths in the file's unit; its points follow from the layout and are not restated. Its identity is of its points, as a target given by them: the layout is not hashed.

An identity does not change when the file format does. An observation names the rig that defines its targets by content_id: the SHA-256 of a canonical tree built from the in-memory rig, serialized by RFC 8785 (JCS, its number formatting), under the current names, specified once in VisualGeometryToolkitIO's docs, in base units (each length normalised to the Float64 its shortest round-tripping decimal in METRES reads back to), its targets in a fixed order, with no schema version and no envelope provenance. A rig, every file written from it in any unit, and a migration of that file to the next record version have one identity; a test pins it for a rig of every target kind.

A published file keeps loading. Every earlier schema version a published file holds is READ, and only the current version is written: a reader that meets an older version migrates it in memory, version by version, to the current record (VisualGeometryToolkitIO's migrate_schema steps, run by migrate_record; a schema's owner adds the steps of its own kinds), silently. The reason is the published data: a schema change must never force an archive on Zenodo or Hugging Face to be uploaded again. Rewriting a file once (vgio_migrate) is optional, and runs the same steps. A version is refused only where no published file holds it, or where it lacks what the current record needs (transforms v1/v2's free-form frame names).

Persisted rotations are quaternions. A rotation is stored as a Hamilton quaternion (w-x-y-z, w ≥ 0; a stored w < 0 is refused on read, never flipped) and READ AS ONE: the reader builds a EuclideanMap{3, Float64, QuatRotation{Float64}, …} from the stored unit quaternion without renormalizing it, so file → value → file is exact and a value read from a file round-trips bit for bit. The writer accepts any Rotation: a QuatRotation is written as it is (its sign made w ≥ 0 by exact negation, the same rotation); any other — a RotMatrix an estimator computed — is converted to a quaternion on write and reads back as a QuatRotation-valued map (equal to the stored matrix to the conversion's rounding). Code that consumes a persisted pose therefore takes any Rotation, never a RotMatrix specifically; one that needs a matrix converts explicitly.

Derivatives and Jacobians

One word, jacobian; the suffix says what comes back. deriv survives only where an upstream package owns the name.

FormMeaningExample
transform_deriv(f, x), transform_deriv_params(f, x)a map's Jacobian w.r.t. input / parametersCoordinateTransformations' API, kept
<f>_with_jacobian(value, J)backproject_with_jacobian(camera, u)
<f>_with_jacobiansvalue plus several, documentedproject_with_jacobians(lens, X) -> (u, J_X, J_p)
constraint_with_jacobians(g, G_x, G_θ) for an implicit constraintconstraint_with_jacobians(E, f₁, f₂; M₁, M₂)
constraint_jacobiansthe Jacobians aloneconstraint_jacobians(M, v, cs)
<thing>_jacobianthe Jacobian alone, named after what it differentiatesretract_jacobian, action_jacobian, observation_gradient_jacobian (∂G_x/∂θ of a constraint)

Each docstring names its Ceres or GTSAM equivalent, and says so explicitly where they disagree:

  • retract_jacobian(ch, v) is ∂vec(retract(ch, v))/∂v — Ceres' Manifold::PlusJacobian. An inverse form would be local_coordinates_jacobian (Ceres' MinusJacobian).
  • left_jacobian(SO3, ω) is Sophus' SO3::leftJacobian. GTSAM's Rot3::ExpmapDerivative is its transpose, the right Jacobian, because GTSAM retracts by right multiplication — measured here, not assumed (left identity residual 5.8e-13; the right identity fails at 1.8e-6).
  • exp_so3/log_so3/exp_se3/log_se3 keep their names: Manifolds.jl's exp(M, p, X) is the base-point exponential while ours is the Lie-group one (exp_lie(G, X)), so exp(SO3, ω) would read as the base-point form with the point dropped.
  • backproject keeps its name: GTSAM's nearest counterpart, Cal3_S2::calibrate, collides with our task verb calibrate (estimating the camera).
  • Implicit constraints follow Förstner's A = ∂g/∂θ, B = ∂g/∂x, since neither Ceres nor GTSAM has the concept. Their Sampson residual has ONE implementation (VisualGeometryToolkitCore's base/sampson.jl): the residual, sampson_residual_with_jacobian's exact model row (from G_θ and the model's observation_gradient_jacobian) and the √w rows over a set are built there and nowhere else; a frozen whitening is whitening_factor.

The stack's Levenberg–Marquardt protocol (VisualGeometryToolkitCore lm_core.jl) is residual_rows! and residual_jacobian_rows!, mirroring Ceres' CostFunction::Evaluate; its driver is NLLSsolver (lmsolve_chart!, one adapter), and the protocol is independent of the driver. They are deliberately NOT named whitened_residuals!: the RANSAC protocol's whitened_residuals!(r, problem, model) is unweighted and in data units, while LM rows are √w-weighted in chart coordinates. One name for two currencies is what the frame rule forbids, so the currency is in the name.

Units

Units are FlexUnits.jl: no stack quantity type or dimension set. The stack's unit code lives in VisualGeometryToolkitCore's src/base/units.jl: its own unit registry, StackUnits, built with FlexUnits' RegistryTools exactly as FlexUnits documents for a registry of another transform type — a PermanentDict of Units{Dimensions{FixRat32}, AffineTransform{Rational{Int}}} filled by register_unit! and exposed by @generate_registry_exports — the unit constants, base_quantity, and the stack's one guarded parse. Parsing (ud"…", uparse, qparse), printing, conversion and dimension errors are FlexUnits'.

Scales are exact rationals. Every registered scale is written as an exact ratio of base units and handed to register_unit! as data: mm = 1//1000, cm = 1//100, μm = 1//1000000, km = 1000, inch = 127//5000 (25.4 mm), ft = 12 inch, pt = inch/72, px and pd dimensionless with scale 1, dpi = px/inch, and the SI base units (m, kg, s, A, K, cd, mol) at 1. The registry holds those and nothing else. FlexUnits' registry_defaults! is not called: in a Rational registry it converts its Float64 scales to their binary values (Rational(0.001) = 1152921504606847//1152921504606846976). So a length built from an integer is exact — 142mm is 71//500 m, whose Float64 is the decimal-nearest 0.142 (FlexUnits' Float64 mm gave 0.14200000000000002) — and a float literal is FlexUnits' promotion, 142.0mm = 142.0 * 0.001, correctly rounded since FlexUnits 0.6.16 (Deduction42/FlexUnits.jl#131: a float times a rational scale as x * num / den) — which rounds the BINARY value of the float, so a decimal read from text can still land one ulp from the decimal's nearest double; VisualGeometryToolkitIO's read_length/write_length (decimal-exact, by exponent shift) remain the stack's one documented exception for that case. An exact quantity compares exactly: ustrip(m, 142mm) == 0.142 is false (the double 0.142 is not 71/500), while ustrip(m, base_quantity(142mm)) === 0.142.

The layering: units at the boundary, plain numbers inside

Units live at the public boundary and plain numbers live inside. A value moves from one to the other where the frames rule already conditions data: once at an estimator's entry and once at its exit.

LayerHolds
Public values (held by users, handed between packages, persisted): EuclideanMap translations, PlanarTarget and PointTarget points, Rig target poses, Trajectory, TargetPoses, a 3D TargetObservations, sensor and board sizesFlexUnits quantities for every physical length, in metres; a dimensionless quantity where a public value has no unit
IOa file declares its length_unit; a reader returns quantities, and a writer writes numbers in the declared unit
Estimator, renderer and solver entry and exit (fit_*, find_*, refine_*, a one-step algorithm's public form)one strip into the private working unit at entry, one reattachment at exit
Inner loop (minimal kernels, *_step updates, ! workspace forms, RANSAC internals, LM rows and charts, GPU kernels)plain numbers of eltype T (Float64, Float32, a ForwardDiff Dual)

Pixel coordinates keep their frame tags, because PixelFrame is their unit. px quantities stay for sizes and densities.

A physical value has two forms, and both are bits types:

FormElement typeWhere
base form, in metresQuantity{T, Dimensions{FixRat32}}public values: poses, points, sizes
plainT (Float64, Float32, Dual)inner loops and GPU kernels

The plain form of a value is the whole value's ustrip(u, X), for a pose, a target, a rig or a capture. It is never a value rebuilt from stripped components. A unit is attached by construction, EuclideanMap(R, t .* m), or by with_unit(X, u), which reads a plain value — a map included, its rotation kept exactly — as being in u. One exception is forced by its type: NoncentralSplineModel holds its line points as plain metres, because its spline of SVector{3, T} controls cannot hold quantities, and its docstring says so. Because a plain extrinsics is "in the caller's unit", a PosedCamera of that model refuses one (ArgumentError, naming t .* m); its plain form is ustrip(m, camera) only.

Units are allowed in a type but not required by it, so the layering is held at four points:

  1. VisualGeometryToolkitIO's readers always return quantities, through read_length.
  2. Its writers refuse a plain translation or point for a physical kind (rig, trajectory, target_poses, transforms, target, a 3D observations, and the object points of correspondences). The ArgumentError names the fix, t .* m.
  3. A public estimator that takes physical values strips them in metres at entry (ustrip(m, x)) and reattaches the unit at exit, so unitful input gives unitful output. Plain input gives plain output, in the caller's own unit, as before units: estimators are not where plain data is refused, the IO writers of point 2 are. A model that cannot record a unit — a plane homography, a radial or division camera, the up-to-scale relative pose of refine_relative_pose — refuses unitful input instead, with an ArgumentError naming the fix (strip the plane points in metres, the unit the stack's boundaries use). This supersedes the plan's earlier "a plain physical input is refused" (ruling of 2026-10-25, Task 9): refusing plain input would break every existing caller, so a later audit must not "fix" the code back to it.
  4. check_naming rule R11 flags a ustrip(…) fed into an EuclideanMap or a target constructor, and a power-of-ten scale in a definition that names a file's length_unit. Unit names are read only by parse_unit.

A pixel is dimensionless. FlexUnits' dimensions are the seven SI ones, so px cannot be told from rad or a ratio, and 3px + 1.0 is allowed. Pixel coordinates keep their safety through PixelFrame tags; only pixel-sized quantities lose it. A function that takes a pixel-sized quantity accepts a plain number or a dimensionless quantity, and refuses a length with DimensionError.

Unit names from outside (a file, a QR code, Python) are read with parse_unit(s) — one registered symbol, or with compound = true a product, quotient or power of them ("mm/px") — and written with unit_string(u), FlexUnits' printing of a registered unit. uparse evaluates what it parses, so nothing else calls it. A dimension check is dimension(q) == dimension(u). check_naming enforces this everywhere (rule R11): no package outside the units module calls uparse/qparse/register_unit, builds a unit or a registry (Units, AffineTransform, PermanentDict, register_unit!, registry_defaults!), reaches into the registry, writes a table of unit names, converts with convert(Quantity{…}, q) or a bare ubase(q) in source (base_quantity(q)), branches x isa Quantity ? ustrip(…) (strip_or_bare(u, x)) or writes a conversion factor (25.4, 0.0254, 304.8, 0.3048, 127//5000); what FlexUnits lacks, or arithmetic that must stay bit-identical to stored values, is allowed by file with its reason (_UNIT_RULE_ALLOW). The one hand-rolled conversion is VisualGeometryToolkitIO's read_length (below), until FlexUnits rounds a rational scale once.

Values are stored in base units. A length is held in metres, a pixel count as a plain dimensionless value, so a bare ustrip(q) of a length is in metres. Code that wants a unit names it: ustrip(mm, q), ustrip(px, q), ustrip(dpi, q). A bare ustrip is correct only on a pixel quantity or a dimensionless ratio. Feature geometry (Disc, Circle, …) strips through one boundary that names its units: pixels in px, lengths in mm.

Dynamic quantities in structs. The exported constants (px, mm, cm, μm, m, inch, ft, pt, dpi) are the registry's dynamic units, so 10.0mm, 3.0px and 300.0dpi share FlexUnits' base-form type Quantity{Float64, Dimensions{FixRat32}}, a bits type; 10mm (an integer times an exact scale) is a Quantity{Rational{Int}, …} until it is stored. A struct that holds quantities stores the Float64 form, named once in VisualGeometryToolkitCore as the computed constant BaseQuantity = typeof(ubase(1.0mm)) — FlexUnits' stored form, not a stack quantity type: no signature takes it, every boundary takes any Quantity — and needs no unit type parameter, and a mixed-dimension container is concrete. Static units (u"…") are not used in struct fields.

Boundaries accept any quantity and convert. A quantity combined with a unit keeps that unit (5.86μm / px is a Quantity{Float64, Units{…}}), and uconvert returns that form too. Public signatures therefore take FlexUnits' Quantity, and a constructor that stores one converts it with base_quantity(q) = float(ubase(q)) (base_quantity.(s) for a Size2, which FlexUnits broadcasts over as it does ustrip.(u, s)), or by converting into a BaseQuantity field. A Rational never reaches a struct field or a kernel: the stack's strips (strip_metres, strip_or_bare) return floats. A pixel–length crossing is a product with a density (q * 300dpi is in px); there is no density-taking conversion verb. A dimension mismatch raises DimensionError (arithmetic) or ConversionError (ustrip/uconvert into a unit of the wrong dimension). Angles are plain radians.

Files keep human units

Memory holds metres. A file holds the unit that reads naturally for its quantity: focal lengths and board sizes in mm, pixel pitch in μm, room-scale poses in m. The file declares its unit, once per file (length_unit) or per field. VisualGeometryToolkitIO's default file unit is mm.

Never convert a file length by multiplying by a scale. The rounding comes from the multiplication, not from metres. 62.8 * 0.001 is 0.0628, but 0.0628 / 0.001 is 62.79999999999999, and the same happens for a few percent of typical millimetre values. VisualGeometryToolkitIO converts every file length with one pair, read_length(x, u) and write_length(q, u). For a power-of-ten unit (mm, cm, μm, km) the pair shifts the decimal exponent, which is one correctly rounded parse. For another unit (inch, ft, pt) it reads by the unit's scale and writes the shortest decimal that reads back to the identical value. A side benefit is that memory values are correctly rounded: 13.9 mm reads as 0.0139 m.

What is exact. A file that VisualGeometryToolkitIO wrote is rewritten byte for byte when its numbers have at most 15 significant digits. A full-precision number (16 or 17 digits) may move on its first rewrite, by at most 2 ulp or to its persisted form, and it stays fixed after that. This is inherent to metres in memory and millimetres in files: the two grids differ, so some 17-digit millimetre numbers share their nearest metre value with a neighbour, and one of the two cannot come back; and a length computed in code may have no file number that reads back to it at all (about one metre value in eight in millimetres). A length's PERSISTED form is its metre value rounded to 15 significant digits, the value VisualGeometryToolkitIO's content_id hashes; it is the length's identity. For a length no file number reads back to, the writer writes a number that reads back as its persisted form (up to ~22 ulp from the value), and a check that a length read back is the one written accepts it within 2 ulp or with the same persisted form. An identity follows the value, not the bytes: the content_id of a rig hashes a canonical tree of the loaded value in metres, so a file, the rig loaded from it and the same rig written in another unit name the same targets.

One quantity form. A JSON record holds each quantity as a number in the unit it declares ("units": {"world": "mm", "density": "dpi"}), through the pair above; a record whose lengths need more than one unit declares each further one as a length role ("lengths": {"pattern": "pt"}), each number decimal-exact in its own. The {"value": v, "unit": u} objects that VisualGeometryToolkitFeatures once wrote, and that BlobBoards' older boards hold, are never a current record's: a file of an earlier schema version that holds them is read by its owner's version step for that version, on load (VisualGeometryToolkitIO's migrate_schema and migrate_record), and vgio_migrate rewrites it once if its owner wants it rewritten. Every earlier version a published file holds stays readable, and only the current one is written: a schema change must not force published data (Zenodo, Hugging Face) to be uploaded again.

A float length built in code has already rounded.[62.8, 1.3] .* mm multiplies by the Float64 of the scale when it is stored in metres. 1.3mm is held as 0.0013000000000000002 m, and a file written from it holds the 17-digit text 1.3000000000000003. An integer length is exact (142mm, 210mm). To build an exact length from a decimal in code, read it the way a file is read: VisualGeometryToolkitIO.read_length(1.3, mm).

Covariances follow the same split. In memory a covariance is in base units: for a pose, rad² in the rotation block, m² in the translation block and m·rad in the cross block. On disk it is in the file's declared length_unit, converted by the same decimal shift raised to each entry's power of length (mm² for a translation entry, mm·rad for a cross entry in a millimetre file).

Stack contracts: tags, dependencies and package boundaries

Six rules hold across the stack. Each is a function in VisualGeometryToolkitCore (src/quality/) that a paper project's suite can call directly, and a gate of VisualGeometryToolkitCore.test_quality_gates, which every stack package's quality test runs.

The tag rule. A configuration file names a type by its tag, and every tag is snake_case of the type's name, with the family's declared suffix stripped from its end when present. snake_case has exactly two rules, then lowercases: an _ goes before an uppercase letter that follows a lowercase letter or a digit, and before the last uppercase letter of an uppercase run when a lowercase letter follows it. A digit stays with its letters: FixedS2 → fixed_s2, WelschHQNewton → welsch_hq_newton, StudentTUniform → student_t_uniform, SimFitPP → sim_fit_pp. A tag that selects nothing is spelled none.

One registry.TagRegistry(family, suffix, "tag" => Type, …; retired, paths) is the only structure a config loader may use to map a tag to a type:

const MODEL_UPDATES = TagRegistry(:model_update, "",
    "keep_model" => KeepModel, "weighted_refine" => WeightedRefine, "none" => nothing;
    retired = ("keep" => "keep_model",),
    paths = [r"local_optimization_strategy\.model_update"])

MODEL_UPDATES["keep_model"]        # KeepModel
MODEL_UPDATES["keep"]              # ArgumentError: tag "keep" was renamed "keep_model"
tag_of(MODEL_UPDATES, KeepModel)   # "keep_model" (the writer side)

An unknown tag is an ArgumentError that lists the known ones. paths are the TOML key paths (dotted, an array element [i] counted from 1, matched whole) whose string values are tags of the family. Construction does not check the rule — a package's load never fails on a spelling; tag_rule_violations does.

The dependency allow-list.configs/stack_dependencies.toml in VisualGeometryToolkitCore has one row per project: the stack packages its Project.toml may name in [deps], [weakdeps], [extras] or the test target. A stack package is any key of that table, and a project missing from it is an error. stack_dependency_violations(project_toml_path, name) checks one project. The table lists the published packages and paper projects only; a project kept elsewhere supplies its own row through the extra keyword (a TOML file in the same format, or a dictionary), and the quality gate's stack_extra. An extra row adds a project and never replaces a published one.

Public names only, across a package boundary. A name imported from, or accessed qualified in, another package is public there (ExplicitImports' check_all_explicit_imports_are_public and check_all_qualified_accesses_are_public), and no name reached from another stack package begins with _ (private_name_violations, a source scan).

Extension points are declared. A package adds methods to another stack package's function only when the owner lists it in its CONTRACT_GENERICS (VisualGeometryToolkitCore's: the block-coordinate-descent hooks, the chart-LM row protocol, the RANSAC/LO interface, the minimal-solver backends, the TOML config hooks, the accessors other packages implement for their own types, the parameter protocol, the camera maps project and backproject, convert_eltype, the Manifold protocol with ambient_coordinates, and the plotting stubs). Any stack package may declare its own; contract_generic_violations(M) lists the methods that land elsewhere. Functions owned by Base, Core, the standard library or a package outside the stack are exempt.

Agent instructions carry nothing private. CLAUDE.md, AGENTS.md and the tracked .claude/ files ship with every package. agent_doc_violations(root) checks them against AGENT_DOC_FORBIDDEN (absolute home-directory paths, private keys and access tokens); every stack package's suite runs it with extra = get(ENV, "AGENT_DOC_PATTERNS", nothing), a file of further patterns that release tooling supplies.

One copy of the conventions. This text lives once, in VisualGeometryToolkitCore's conventions/ folder (VisualGeometryToolkitCore.conventions_dir()). A package's CLAUDE.md points at the folder and carries no copy of it; each package's agent-docs test checks that its CLAUDE.md names the folder and holds none of the shared sections.

Gate keywordChecksFunction
tag_rulesevery TagRegistry constant of the package obeys the tag ruletag_rule_violations
tag_tablesno Dict{String, <:Type} table or "tag" => SomeType pair outside a TagRegistry(...) call in src/ (a heuristic scan; comments, docstrings and Base/Core converter types such as "cutoff" => Float32 are skipped)tag_table_violations
config_tagsevery string at a registry's paths in the TOML files under config_dirs (default: the package's configs/) is a current tagconfig_tag_violations
stack_dependenciesthe Project.toml against the package's row (stack_name)stack_dependency_violations
public_namesthe two ExplicitImports public checks and the private-name scanprivate_name_violations
contract_genericsmethods on other stack packages' functions land on their contractscontract_generic_violations

Each gate takes a mode: :enforce (@test that it finds nothing), :report (each finding logged with @warn and the gate recorded as broken; a gate that finds nothing is an ordinary test) or :off. Every gate defaults to :report until the stack conforms; a package turns one to :enforce by supplying the keyword.

See also