Detection
VisualGeometryToolkitFeatures.VisualGeometryToolkitFeatures — Module
VisualGeometryToolkitFeaturesFeature detection and descriptor matching.
- Detection: ASIFT (IMAS-25) on KernelAbstractions.
asift_detectruns on the CPU or any KernelAbstractions GPU backend and returnsDetections (a frame plus metadata). VisualGeometryToolkitIO writes and reads them as JSON (json_value,from_json). The image circle of a lens, measured from photographs (detect_image_circle: the rim's log-brightness drops on log-polar profiles here, the RANSAC fit through the VisualGeometryToolkitEstimation extension). - Matching: nearest-neighbour descriptor matching, one KernelAbstractions implementation on the CPU or any GPU backend (
nn_match,match), and a stacked matcher over many galleries (nn_match_stacked). - LoMa: the learned DeDoDe descriptors and LoMa matchers, from the authors' ONNX exports (
LoMaDescriptor,LoMaMatcher). - Camera RAW: the stack's one RAW path.
load_rawdecodes a Bayer RAW file (LibRaw) into VisualGeometryToolkitCore'sRawMosaicand its exact linear values on the 2×2 Bayer cells (Core'sLinearRaw: the green-cell image for geometry and radiometry, or R G B), with saturation flags, true sample positions and the exact frame map onto the raw (Core'sraw_frame_map); a display-only demosaic for figures, of a file or a mosaic. - Extensions: JLD2 backs
FileCache; Makie draws detections and correspondences; LibRaw_jll is the RAW decoder.
The detection type and its polarity helpers, the ASIFT detector and its plans, the detection covariance, the Gaussian scale space and gradient, the MSER, Förstner and SFOP detectors, the spatially spread selection of detections (SSC), and the image circle of a lens measured from photographs. The image-input helpers and the PNG and RAW readers are on Image Input.
VisualGeometryToolkitCore.Primitives.Disc — Method
Disc(d::Detection) -> DiscExtract or convert to a Disc frame (uses characteristic_scale for anisotropic).
VisualGeometryToolkitCore.Primitives.Ellipse — Method
Ellipse(d::Detection; cutoff=1.0) -> EllipseConvert to an ellipse at the cutoff·σ contour (via region).
VisualGeometryToolkitCore.Primitives.EllipticFrame — Method
EllipticFrame(d::Detection) -> EllipticFrameExtract or convert to an EllipticFrame.
VisualGeometryToolkitFeatures.Detection — Type
Detection{F<:FeatureFrame, M}Type alias for Attributed{F, M} — a detection result composing a geometric frame with detector-specific metadata.
The frame type F determines the geometry:
Detection{Disc{T}, M}- isotropic detection (3 DOF)Detection{OrientedDisc{T}, M}- oriented isotropic detection (4 DOF)Detection{EllipticFrame{T}, M}- anisotropic detection (5 DOF)Detection{AffineFrame{T}, M}- full affine detection (6 DOF)Detection{<:Uncertain{F}, M}- a frameFwith its covariance (VisualGeometryToolkitCore'sUncertain(frame, Σ)), fromasift_detect(…; uncertainty = FeatureCovariance()):cov(d)is the covariance,feature_frame(d)the bare frame,geom(d)the detection without it, and the properties below read as on the bare detection
Type Parameters
F<:FeatureFrame: The geometric frame typeM: Metadata, aNamedTuple. ASIFT emits(response, edge_score, laplacian_scale_score, warp_idx);laplacian_scale_scoreis the polarity fieldis_dark/is_lightread (> 0 dark, < 0 light).
Fields (from Attributed)
value::F: The geometric frame (center + shape)attributes::M: Detector-specific scores and information
Coordinates
The frame's center is (x, y) in COLMAP pixel-centre coordinates, the convention of every detector in this package: x along the image columns, y along the rows, the top-left pixel's centre at (0.5, 0.5). The pixel img[r, c] has its centre at (c − 0.5, r − 0.5). asift_detect documents its measured sub-pixel accuracy.
Property Access
Properties are forwarded from frame and metadata:
d.frame,d.value→ the geometric framed.metadata,d.attributes→ the metadatad.center→frame.centerd.σ→ scale from frame (exact for Disc, characteristic_scale for anisotropic)d.Σ→ shape matrix (blob shape, not the position covariance)d.response,d.edge_score, etc. → forwarded from metadata
Example
julia> using VisualGeometryToolkitFeatures, VisualGeometryToolkitCore
julia> det = Detection(Disc(Point2(100.0, 200.0), 5.0), (response = 0.05, laplacian_scale_score = -0.1));
julia> Tuple(det.center), det.σ, det.response # from the frame and the metadata
((100.0, 200.0), 5.0, 0.05)
julia> is_light(det)
trueSee also: Disc, EllipticFrame, VisualGeometryToolkitCore's Attributed
VisualGeometryToolkitFeatures.HasPolarity — Type
HasPolarity()The HasPolarityTrait of detection metadata that records a polarity: a NamedTuple with a laplacian_scale_score field, positive for a dark blob and negative for a light one. is_dark and is_light read that field's sign. Public, not exported.
VisualGeometryToolkitFeatures.HasPolarityTrait — Type
HasPolarityTraitHoly trait for metadata types indicating whether they contain polarity information: HasPolarity or NoPolarity, as polarity_trait returns it. Public, not exported.
VisualGeometryToolkitFeatures.NoPolarity — Type
NoPolarity()The HasPolarityTrait of detection metadata that records no polarity. is_dark and is_light are both false for such a detection. Public, not exported.
VisualGeometryToolkitCore.propagate — Method
propagate(m, d::Detection{<:Uncertain}) -> UncertainThe uncertain centre of a detection through the map m (a frame map onto a camera's pixels, say): VisualGeometryToolkitCore's one detection → point observation conversion center(d) (its centre with the position block of its frame's covariance), carried through m by Core's propagate(m, ::Uncertain).
VisualGeometryToolkitFeatures.is_dark — Method
is_dark(d::Detection) -> BoolWhether d is a dark blob on a lighter background: its metadata records a polarity (see polarity_trait) and its laplacian_scale_score is positive. false for a detection without a recorded polarity.
julia> using VisualGeometryToolkitFeatures, VisualGeometryToolkitCore
julia> is_dark(Detection(Disc(Point2(1.0, 2.0), 3.0), 0.5; polarity = :dark))
true
julia> is_dark(Detection(Disc(Point2(1.0, 2.0), 3.0), 0.5)) # no polarity recorded
falseSee also is_light.
VisualGeometryToolkitFeatures.is_light — Method
is_light(d::Detection) -> BoolWhether d is a light blob on a darker background: its metadata records a polarity (see polarity_trait) and its laplacian_scale_score is negative. false for a detection without a recorded polarity. ASIFT detections record one, from the sign of the DoG response.
julia> using VisualGeometryToolkitFeatures, VisualGeometryToolkitCore
julia> d = Detection(Disc(Point2(1.0, 2.0), 3.0), 0.5; polarity = :light);
julia> is_light(d), is_dark(d)
(true, false)See also is_dark.
VisualGeometryToolkitFeatures.polarity_trait — Function
polarity_trait(::Type{M}) -> HasPolarityTrait
polarity_trait(d::Detection) -> HasPolarityTraitWhether metadata of type M, or the metadata of the detection d, records a polarity: HasPolarity for a NamedTuple with a laplacian_scale_score field, NoPolarity otherwise. A detector whose metadata is another type opts in by adding a method for its type, and must then give it a laplacian_scale_score property. Public, not exported.
julia> using VisualGeometryToolkitFeatures, VisualGeometryToolkitCore
julia> import VisualGeometryToolkitFeatures as VgtkFeatures
julia> VgtkFeatures.polarity_trait(Detection(Disc(Point2(1.0, 2.0), 3.0), (response = 1.0,)))
VisualGeometryToolkitFeatures.NoPolarity()
julia> VgtkFeatures.polarity_trait(typeof((response = 1.0, laplacian_scale_score = -1.0)))
VisualGeometryToolkitFeatures.HasPolarity()VisualGeometryToolkitFeatures.MserRegion — Type
MserRegionOne maximally stable extremal region, as detect_mser returns it.
Fields
seed::CartesianIndex{2}: the region's reference pixel, a pixel at the region's level (the tree nodeMaxTreechose; VLFeat may pick another pixel of the same plateau). A VLFeat seeds(0-based linear index into the column-majorH × Wimage given withdims = (H, W)) is the pixelCartesianIndices(img)[s + 1], and ours is VLFeat's indexLinearIndices(img)[seed] - 1.level::UInt8: the region's level in the input image: a:darkregion is the 8-connected component ofimg .<= levelholdingseed, a:lightone that ofimg .>= level.img[seed] == level.area::Int: its number of pixels.variation::Float32: VLFeat's (in)stability(|R₊Δ| − |R|) / |R|.polarity::Symbol::darkor:light.
The seed and level determine the region, so its pixels and boundary are not stored (nested regions would hold every pixel many times over): they are formed on demand from the image the region was detected in — region_mask(region, img) the pixels as a mask (VLFeat's vl_erfill), boundary_contours(region, img) their boundary as closed polygons. is_dark and is_light read the polarity.
VisualGeometryToolkitFeatures.boundary_contours — Method
boundary_contours(pixels) -> Vector{Vector{Point2{Float64}}}
boundary_contours(region::MserRegion, img) -> Vector{Vector{Point2{Float64}}}The boundary of a set of pixels (CartesianIndex{2}(row, column), a Bool mask, or an MSER of img — its region_mask), traced along the pixel edges: one closed polygon per boundary curve (the last vertex joins the first), vertices only where the boundary turns, with x along the columns and y along the rows in the COLMAP pixel origin (conventions/pixel-origins.md), so the pixel (r, c) is the square [c − 1, c] × [r − 1, r] and every vertex is a pixel corner with integer coordinates. The polygons are sorted by signed shoelace area, descending: the outer boundaries first (positive, clockwise on screen), the largest first, then the holes (negative), the smallest first and the largest last. Diagonal neighbours are connected (8-connectivity, the connectivity of an MSER), so where two pixels of the set meet at a corner one curve runs through it; holes are 4-connected.
VisualGeometryToolkitFeatures.detect_mser — Method
detect_mser(img; delta = 5, min_area = 3 / length(img), max_area = 0.75,
max_variation = 0.25, min_diversity = 0.2,
bright_on_dark = true, dark_on_bright = true) -> Vector{MserRegion}The maximally stable extremal regions (MSER; Matas, Chum, Urban and Pajdla, "Robust wide baseline stereo from maximally stable extremal regions", BMVC
- of the 8-bit gray image
img, selected as VLFeat'svl_mserselects
them (Andrea Vedaldi and Brian Fulkerson, BSD license; src/mser.jl carries the notice). VLFeat's stability criterion is its own, simpler than the paper's.
The extremal regions are the 8-connected components of the level sets img .<= l, arranged in a tree by ImageMorphology's MaxTree. The variation of a region R at level l is (|R₊Δ| − |R|) / |R|, where R₊Δ is the largest region containing R at a level at most l + delta. Of a region and its parent region exactly one level up, the one of larger variation (on a tie, the child) is not maximally stable; the root never is. The survivors are then filtered, largest first:
max_variation: a region of variation>= max_variationis dropped;max_area,min_area: FRACTIONS oflength(img); a region larger thanmax_area * length(img)or smaller thanmin_area * length(img)pixels is dropped (the defaultmin_areais three pixels);min_diversity: a region whose area differs from that of its nearest surviving enclosing region by less thanmin_diversitytimes the latter's area is a duplicate and is dropped (min_diversity = 1turns this off).
An image of fewer than three pixels has no region (VLFeat returns none): MserRegion[], the area fractions not checked.
dark_on_bright returns the dark regions (polarity = :dark) and bright_on_dark the light ones (:light), the components of img .>= l: VLFeat runs the same on 255 .- img, whose <= level sets these are. Dark regions come first; each polarity is ordered as VLFeat orders it, by level (darkest first, resp. brightest first), then by seed.
Input
A matrix of UInt8, N0f8 or Gray{N0f8}: VLFeat's vl_mser_pix is a byte, and these hold its 256 levels exactly. Any other element type is an ArgumentError: quantize explicitly (e.g. gray_u8), so the levels the regions are cut at are the caller's choice.
julia> using VisualGeometryToolkitFeatures
julia> using VisualGeometryToolkitFeatures: boundary_contours
julia> img = fill(0xc8, 40, 60); img[11:20, 21:35] .= 0x28; # a dark rectangle
julia> r = only(detect_mser(img; bright_on_dark = false));
julia> r.polarity, r.area, Tuple.(only(boundary_contours(r, img)))
(:dark, 150, [(20.0, 10.0), (35.0, 10.0), (35.0, 20.0), (20.0, 20.0)])VisualGeometryToolkitFeatures.region_mask — Method
region_mask(region::MserRegion, img) -> BitMatrixThe pixels of region, an MSER of img, as a mask the size of img: the 8-connected component of img .<= region.level (:dark; .>= for :light) holding region.seed, filled from the seed as VLFeat's vl_erfill fills it. findall of it lists the pixels in ascending linear order. img must be the image the region was detected in: a seed outside it, a level other than img[seed] or a fill of another area than region.area is an ArgumentError.
VisualGeometryToolkitFeatures.FoerstnerTests — Type
FoerstnerTests()A table of the Förstner F-test's critical values (detect_foerstner's f_tests keyword): each window half-width's pair (the corner's and the circular point's critical F at the significance level, from the residual sums' effective degrees of freedom) is computed on its first use and read by every later call. The values depend on the configuration only — the gradient taps (derivative, derivative_sigma), the half-width ρ, the pixel noise's structure (white input pixels, or the photosites through a mosaic's bilinear green image) and significance_level — and each entry is keyed by all four, so one table serves calls of different settings and the rows are bit-identical to a call without one. The caller makes one per detect configuration and gives it to each of its calls (a set of photographs: the host computation was most of a seeded call's time). Safe to share between threads.
VisualGeometryToolkitFeatures.detect_foerstner — Method
detect_foerstner([T = Float32], img, seeds::AbstractVector{<:Detection}; device = default_device(),
noise = nothing, derivative = :gaussian2d, derivative_sigma = 1.0,
significance_level = 0.999, polarity = :none, f_tests = nothing) -> StructArray{<:Detection}
detect_foerstner([T = Float32], L::LinearRaw, seeds; sensor_model::SensorModel = SensorModel(), kwargs...)The Förstner operator's window estimation (steps 4–6 of the method above, its doubled resolution, its kernels) at GIVEN windows: one window per seed, centred on the doubled-grid pixel nearest the seed's centre, its integration scale the seed's characteristic scale σ (input pixels; a detect_sfop row's selected σ). The box window has the standard deviation σ, the estimation window the half-width ⌈√(12(2σ)² + 1)/2⌉ + 1 doubled pixels, as integration_sigma = σ would give: SFOP's N(σ) = 12σ² + 1 is the pixel count of the box of the same energy as its Gaussian G_σ. The image-wide window search (step 3) is not run.
On a RAW's green cells (L::LinearRaw, positions on the cell grid; seeds from detect_sfop(L)) the frame's sensor_model gives the covariance as for the image-wide call on L.
The seeds are detections of a frame with a scale (a Disc, an OrientedDisc, an EllipticFrame or an AffineFrame, its characteristic scale σ; each also with its covariance); a point has no window (an ArgumentError naming its frame). The keywords are the ones above: the image-wide search's (method, noise_sigma, integration, integration_sigma, precision_threshold, roundness_threshold) are not read at given windows and are refused (ArgumentError, naming them).
polarity = :dark | :light keeps only the seeds whose blob has that sign, detect_sfop's test: the scale-normalised Laplacian of Gaussian at the seed's σ, its sign significant at significance_level under the pixel noise (noise on a matrix, the gain-carrying sensor_model on a LinearRaw; with none known the call is refused). :none (the default) keeps every seed.
f_tests: a FoerstnerTests table the F-test's critical values are read from (each window half-width's computed once, on its first use): give one table to every call of a configuration (a set of photographs), else each call computes them again on the host. The rows are the same either way.
Returns the rows of detect_foerstner(img) (label, frame and covariance, window, F; the F-test of each window at its own half-width's degrees of freedom), one per kept seed whose window lies inside the doubled image, in the order of seeds. Two seeds of one blob (a dot SFOP sees at two scales) give two rows: duplicates are the consumer's to resolve (its own geometry, e.g. an assignment to a target).
VisualGeometryToolkitFeatures.detect_foerstner — Method
detect_foerstner([T = Float32], img; device = default_device(),
method = :foerstner, noise = nothing, noise_sigma = something(noise, 2.0),
derivative = :gaussian2d, integration = :box,
derivative_sigma = 1.0, integration_sigma = 1.41 * derivative_sigma,
precision_threshold = 0.5, roundness_threshold = 0.3,
significance_level = 0.999, f_tests = nothing) -> StructArray{<:Detection}
detect_foerstner([T = Float32], L::LinearRaw; sensor_model::SensorModel = SensorModel(), kwargs...)The Förstner interest operator (Förstner and Gülch, 1987): the junction (corner) points and circular points of a grey image at sub-pixel precision, each with its covariance, after the reference implementation ip_fop.m (Luxen, 2005), with Köthe's doubled resolution and window criterion (Köthe, 2003).
Returns one row per optimal window, a StructArray of Detection(Uncertain(PointFrame(x), Σ), (; label, window, F, covariance_status)) (VisualGeometryToolkitCore's PointFrame: a position, no scale):
label::corner(a junction point),:circular(Förstner & Gülch's circular point: the centre of a rotationally symmetric pattern, with no size estimate) or:unknown(the significance test assigned neither);- the frame: the corner or circular point's position and its 2×2 covariance (px²,
(x, y)order); for an:unknownrow the window centre with a NaN covariance (the reference returns no estimate for it); window: the window centre, aPoint2;F: the classification statisticΩ_corner / Ω_circular;covariance_status: how the frame's covariance was obtained, or why there is none: the image noise propagated through the detector (below) —:propagatedfor a given σₙ,:propagated_measured/:propagated_estimated/:propagated_specifiedfor a RAW under its frame'sSensorModel, by the source of its Poisson–Gaussian noise —:a_posteriori(the reference'sΩ/redundancy · N⁻¹, when no noise is known), or:unclassifiedon an:unknownrow (no estimate, a NaN covariance).
filter(d -> d.label === :corner, points) are the corners; Ellipse(d.value; sigma = 1) is a point's 1σ error ellipse (ip_errell.m's default).
img is any grey image the package's detectors take, read through gray_f32 as grey values in [0, 1] (an 8-bit image as value/255, as MATLAB's im2double); it is not stretched to its own range. The image-wide steps run on device, any KernelAbstractions backend, in precision T (the reference computes in Float64; supply Float64 for that): Float32 or Float64, any other refused (an ArgumentError): in Float16 the squared gradient's window sums would accumulate in it. A Float16-stored image (LinearRaw(Float16, …)) is read in T.
Keywords (defaults: the values the reference's CODE runs with)
method::foerstner— windows where the roundness4 det/tr² > roundness_thresholdand the precisiondet/tr > (σₙ / precision_threshold)², ranked bydet/tr;:koethe— windows whose precision test succeeds, ranked by the smaller eigenvalue.noise: the image noise the covariance is propagated from: the standard deviation σₙ of iid pixel noise, in grey levels of an 8-bit image (0–255, asnoise_sigma). The covariance is thenσₙ² J Jᵀ(σₙon the[0, 1]image),Jthe exact first-order derivative of the estimate with respect to every input pixel, through the detector's own doubling, derivative taps, squared gradient and window estimate (step 6).nothing: no noise is known, and the covariance is the reference's a-posterioriΩ/redundancy · N⁻¹(step 4), markedcovariance_status = :a_posteriori; that one is not calibrated (it carries the model misfit, which does not scale with the noise).sensor_model(a RAW's green cells,L::LinearRaw, read through VisualGeometryToolkitCore'sgreen_plane, positions on the cell grid): the frame's resolvedSensorModel(sensor_model), the oneLwas linearised with. Under a Poisson–Gaussian noise (a gain)Lcarries each value's variance (Core's one law), and each pixel's variance weighs its column,Σ = J diag(var) Jᵀ; read noise alone, a relative noise or none leaves the a-posteriori covariance (the owner's ruling: only a gain-carrying model is fed).noise_sigma: the noise level of the PRECISION TEST (step 3), in grey levels of an 8-bit image (0–255), as the reference's for auint8image;σₙ = noise_sigma / 255on the[0, 1]image. It defaults tonoisewhen that is given, else to the reference's 2. It does not enter the covariance.derivative::gaussian2d(a Gaussian derivative smoothed across by a Gaussian of the same σ) or:gaussian1d(the derivative alone).integration::box(a box whose standard deviation isintegration_sigma) or:gaussian.derivative_sigma,integration_sigma: in input pixels.precision_threshold: the largest accepted point standard deviation, input px.roundness_threshold: the smallest accepted roundness (:foerstneronly).significance_level: of the F-test that labels a point corner or circular.f_tests: aFoerstnerTeststable the F-test's critical values are read from, one per configuration, shared by its calls;nothing(the default) computes them in each call. The rows are the same either way.
Algorithm (ip_fop.m, step by step)
- The image is doubled bilinearly (the package's one 2× upsampling, ASIFT's) and every scale and threshold doubled with it (lines 170–180).
- The gradient by correlation with the derivative taps (radius
⌈3.3σ⌉, zero padding, asimfilter:gaussian_gradient), its squares and product (the squared gradient), and their integration by the box (width2⌈√(12σ² + 1)/2⌉ + 1) or the Gaussian (radius⌈3.3σ⌉), zero padded (lines 184–207, 617–678). - The window measure (
det/tror the smaller eigenvalue) masked by the tests above; its regional maxima (8-connected plateaus higher than every neighbour,imregionalmax), thinned by a second non-maximum suppression over a square of radius2⌈√(12σᵢ² + 1)⌉doubled pixels, away from the border (lines 210–281). - In the
(2ρ + 1)²window around each centre,ρ = ⌈√(12σᵢ² + 1)/2⌉ + 1(σᵢ doubled), the pixels weighed equally (the reference's choice, also for a Gaussian integration): the corner is the least-squares intersection of the lines through every pixel ACROSS its gradient (normal matrixΣ ∇g∇gᵀ), the circular point that of the lines ALONG it (Σ ∇g⊥∇g⊥ᵀ). Each has a weighted residual sumΩ, redundancy(2ρ + 1)² − 2and a-posteriori covarianceΩ/redundancy · N⁻¹(lines 314–423, 440, 449). F = Ω_corner / Ω_circularagainst the F distribution of the two residual sums' EFFECTIVE degrees of freedom(ν_corner, ν_circular), as the reference's(T < F(α)) − (T > F(1 − α))(line 429): a corner below the1 − significance_levelquantile, a circular point above thesignificance_levelquantile, unknown in between.ν = (tr Q)²/tr Q²(Satterthwaite),Ω = Iᵀ Q Ithe residual sum as a quadratic form of the input pixels under white noise, for the window's half-width and the derivative taps, about the window centre (difference 4 below).- With
noisegiven, the covariance is propagated instead (not in the reference): the estimatex̂ = N⁻¹ bmoves, by the implicit-function theorem, byδx̂ = N⁻¹ Σₚ A(p) δ∇g(p)withA(p) w = (vᵀe) R w + v eᵀ R w,v = R∇g,e = p − x̂(Rthe identity for a corner, the 90° rotation for a circular point), the window centre fixed; the gradient is linear in the input, so each input pixelqgives the columnJ[:, q], exact at the image borders, andΣ = Σ_q σₙ² J[:, q] J[:, q]ᵀ. Monte Carlo against each dot's noise-free estimate gives the χ²₂ NEES ≈ 2 and 95 % coverage (Förstner study, Part 2).
Coordinates
COLMAP pixel centres of the input image, (x, y) = (column, row), converted once, at the output. The doubled image's pixel (r′, c′) (1-based) samples the input at 1-based (r′/2, c′/2), so a doubled-grid position is at COLMAP ((c′ − 1)/2, (r′ − 1)/2) in the input, and its covariance is the doubled one /4, reordered to (x, y).
Differences from ip_fop.m
- Undoing the doubling — fixed. The reference doubles with MATLAB's
imresize(g, 2, 'bilinear')(line 172; doubled pixeljsamples inputj/2 + 1/4) but maps positions back withtransf_H,transf_v(lines 340–350, applied at 436, 446, 457), the inverse of an align-corners grid: its points are off by((n + 1)/2 − x)/(2n − 1)(1-based,nthe image size), 0 at the centre, up to 0.25 px toward the centre at the borders. Here the doubling is the package's upsampling (inputj/2, a quarter pixel fromimresize's grid) undone by its own exact inverse. - The circular point's residual sum — kept as the reference has it. Its line 423 takes two of the three terms from the corner model's residuals; the circular model's own residual sum would take all three from its own. Monte Carlo coverage favours the reference's form, so it is kept.
- Defaults — the code's, not the header's. The header states
SIGMA_N = 2.5andSIGMA_DIFF = 0.7(lines 37, 50); the code runs2.0and1.0(lines 521, 524). A documentation conflict, not a fix. - The F-test's degrees of freedom — corrected. The reference tests
FagainstF(redundancy, redundancy), the window's doubled-pixel count less 2, as if every residual were independent. The doubled samples are not: the doubling and the derivative taps correlate them, and each residual sum has the effective degrees of freedomνof step 5, about 1/23 of the pixel count (ν ≈ 87 against 2023 at ρ = 22,derivative_sigma1). At the pixel count the test's level is not the stated one: on white noise about 20 % of the windows were labelled circular and 25 % corner atsignificance_level0.999 (the suite's test), whereνgives at most the stated rate. On the chrome photographs the windows between dots, on line artefacts and on letter strokes (F ≈ 1–3) were labelled circular. The a-posteriori covariance keeps the reference'sΩ/redundancy · N⁻¹(its calibration, NEES ≈ 2, is the Förstner study's).
The docs page gives the evidence for each. The reference warns that with a Gaussian integration the window estimation still weighs the pixels equally; so does this.
Example
julia> using VisualGeometryToolkitFeatures, VisualGeometryToolkitCore
julia> img = [Float32((c - 20.5)^2 + (r - 24.5)^2 < 6.25) for r in 1:48, c in 1:40]; # a dot at pixel (24.5, 20.5)
julia> points = detect_foerstner(img);
julia> only(points).label
:circular
julia> round.(Tuple(only(points).center); digits = 1) # COLMAP: (column − ½, row − ½)
(20.0, 24.0)VisualGeometryToolkitFeatures.gaussian_gradient — Function
gaussian_gradient(img::AbstractMatrix{<:Real}, σ[, mask]; derivative = :gaussian2d) -> (gx, gy)The gradient of img at scale σ (pixels): gx the derivative along the columns (COLMAP x, ∂/∂column), gy along the rows (y, ∂/∂row), each the image's size, in its element type (an integer image's float), on its device (the image's KernelAbstractions backend: a host array on the CPU, a GPU array on its GPU — one kernel for both). Per pixel, so a step of height 1 gives a peak of ≈ 1/(√(2π)σ) per pixel for :gaussian2d.
The derivative of a Gaussian, correlated along one axis: the taps x·exp(−x²/2σ²)/(√(2π)σ³) on radius ⌈3.3σ⌉; along the other axis derivative = :gaussian2d smooths with the unit-sum Gaussian of the same σ and radius, :gaussian1d not at all. Zero padding at the image border (MATLAB's imfilter): the Förstner operator's gradient (detect_foerstner calls this one), bit for bit.
With a mask (an H × W array of weights, 0 outside the support — a fisheye's image circle) it honours gaussian_blur's semantics: the gradient of the NORMALIZED convolution blur(img·mask) / blur(mask) (the blur the across-axis Gaussian), (∂N·D − N·∂D) / D² with N and ∂N of img·mask, D and ∂D of mask; pixels outside the mask do not bleed in, and the gradient is 0 where the mask is zero.
VisualGeometryToolkitFeatures.batched_rows — Method
batched_rows(run_chunk, N::Int, max_batch::Int; retry_single = false) -> Matrix{Float32}Run run_chunk(offset, n) over 1:N in chunks of at most max_batch rows (max_batch <= 0 means one chunk) and assemble the N × D result: the row width D is read off the FIRST chunk's output (any shape with n * D elements), the matrix is allocated lazily then, and each chunk is assigned into its contiguous row slice. With retry_single, a chunk that throws is retried one row at a time (the batch size drops to 1 for the rest of the run) — the recovery an inference session wants when a batch overflows its arena. N == 0 returns an empty 0 × 0 matrix.
Hoisted from BlobBoards, where the ONNX host, ORT device-tensor and two Lux describe backends had each drifted their own copy (2026-09).
VisualGeometryToolkitFeatures.asift_detect — Method
asift_detect(img; device=default_device(), nms=:mxm, window_half=3,
cutoff=3.0, intensity_thr=0.04, edge_thr=10.0,
polarity=:none, band_octaves=0.0, band_reach=1, boundary_K=3.0,
tilts=:imas25, exclusion_mask=nothing, upsample=1, nb_scales=3,
nb_octaves=sift_nb_octaves(W, H; upsample), seed_sigma=1.6,
raw=false, uncertainty=nothing, refinement=nothing)Run the IMAS-25 ASIFT detector on img (H×W; Gray, UInt8, or real-valued in [0,1]) and return Vector{Detection{EllipticFrame}} in COLMAP coords.
Coordinates. A detection's center is (x, y), with x along the image columns and y along the rows, in COLMAP pixel-centre coordinates: the top-left pixel's centre is (0.5, 0.5), so a blob centred on the pixel img[r, c] is returned at (c − 0.5, r − 0.5). Measured on isotropic Gaussian blobs on the CPU (pinned by the test suite):
- a blob centred on a pixel centre comes back exact (within 0.005 px);
- between pixel centres, the quadratic sub-pixel refinement pulls the centre toward the nearest pixel centre, by at most 0.03 px at σ = 4 (−0.007, −0.015, −0.021, −0.027 px at fractions 0.1–0.4, mirrored above 0.5) and 0.06 px for σ from 2 to 8;
- an exact half-pixel centre, where the DoG ties between the two pixels, is refined like any other: the extremum test breaks the tie toward the first pixel, so it carries that side's bias (−0.031 px at σ = 4). A test strict against all 26 neighbours would reject both pixels;
- every detection is refined: extrema are searched on the interior scale slices only (each with a slice above and below), and an extremum whose refinement fails is dropped (
plan.refine_droppedcounts them), never returned on the octave's pixel grid; withuncertainty = FeatureCovariance(), so is a detection whose covariance is singular or NaN (counted there too).
With raw = true the features are in 0-indexed OpenCV coordinates (top-left pixel centre (0, 0)), 0.5 less on both axes.
nms selects the cross-warp non-max suppression: :mxm (centroid window, default), :ellipse (Mahalanobis footprint), or :none (full per-warp feature pool). polarity gates by DoG sign (:dark → response > 0, :light → response < 0, :none → both) BEFORE the cross-warp NMS, so a blob of the other sign never suppresses a kept one. band_octaves > 0 makes the NMS scale-banded: a feature can only be suppressed by features whose geometric-mean scale band ⌊log2(√det S) / band_octaves⌋ is within band_reach of its own, so board-sized extrema and tilted-warp streaks stop erasing the small blobs inside them (see asift_detect!). raw=true returns the ImasFeatureKA vector instead of Detections. device, tilts, upsample, nb_scales, nb_octaves and seed_sigma pick the plan; the others but raw and exclusion_mask go to asift_detect!, which refuses a keyword it does not take.
Scales.upsample (how many times the first octave is the image upsampled 2×; true/false read as 1/0), nb_scales and nb_octaves set the scales the extremum search reads, the interior DoG slices of every octave: see AsiftPlan for the range they span.
Tilts.tilts is a tilt_schedule name or a vector of (t, φ) pairs: :imas25 (the default) the 25 tilts of IMAS-25, :untilted the identity warp alone (plain DoG-SIFT), whose detections localize best (about 0.3 px rms on real panorama frames, against about 0.95 px for a tilted warp's): the schedule for views that differ by a rotation, such as a panorama's. Under :untilted every detection has warp_idx == 1, and they are exactly the identity warp's detections under :imas25.
Exclusion mask.exclusion_mask, an H×W AbstractMatrix{Bool} (true where detection is not allowed: clipped or saturated pixels, a vignetted rim, an overlay), drops every detection whose centre lies within three of its scales (characteristic_scale, the DoG's support) of a masked pixel, so no detection is read off masked pixels. It applies after the cross-warp NMS, to raw features and covariances alike. The detector on a StabilizedImage masks its saturated pixels by this rule.
Determinism. The result is a function of the image and the keywords: the same detections, in the same order, run to run, on one thread or many, launched statically or from inside a threaded region. The extrema pool fills in the order the threads reach it, so before the cross-warp NMS (which breaks ties in its quantized response by pool order) it is sorted into a canonical order: by position, scale and response, then every other field (the covariance's too). The detections come back in that order.
Detection uncertainty.uncertainty = FeatureCovariance() returns each detection's frame as Uncertain(frame, Σ̃), Σ̃ = cov(d) a 3×3 CovMat over (x, y, log σ) for unit iid pixel noise in the detector's intensity units — see FeatureCovariance for the model, the parameterization and the units. With raw = true the result is then (; features, cov). The default nothing runs the detector without it, the kernels compiled exactly as before.
Continuous refinement (opt-in).refinement = ContinuousRefinement() replaces each detection's one-step quadratic refinement, after it, by an iterative one on a continuous model of the DoG response (Core's device LM, in a kernel, on the CPU and any GPU): the same candidates, the half-sample jump of the one-step refinement gone, and with uncertainty = FeatureCovariance() each refined detection's covariance recomputed at its own point from that model's derivatives. A detection whose continuous refinement fails keeps today's. See ContinuousRefinement for the response models; the default refinement = nothing is the detector unchanged.
The plan for (size, device, tilts, upsample, nb_scales, nb_octaves, seed_sigma) is cached (see set_max_cached_plans!) and every call holds a process-wide lock, so concurrent calls are safe but do not run in parallel. On a KA.CPU() device every kernel launch is statically scheduled over the threads, or dynamically when the call runs inside a threaded region (e.g. a Threads.@threads loop over frames); the features are the same either way.
VisualGeometryToolkitFeatures.asift_detections — Method
asift_detections(feats::Vector{ImasFeatureKA}) -> Vector{Detection}
asift_detections(feats, cov) -> Vector{Detection}Convert raw ASIFT features (0-indexed opencv pixel coords, input frame) into Detection{EllipticFrame}s in COLMAP convention (+0.5 on both axes, so the top-left pixel's centre is (0.5, 0.5); see asift_detect). The metadata is (response, edge_score, laplacian_scale_score, warp_idx): laplacian_scale_score = sign(response) (+1 dark, −1 light — the DoG sign convention polarity gates on), which is what is_dark and is_light read; edge_score is a placeholder (always 0). Given the features' covariances (asift_detect!(…; uncertainty = FeatureCovariance())), each frame is Uncertain(frame, Σ̃) (see FeatureCovariance); the +0.5 shift leaves a covariance unchanged.
VisualGeometryToolkitFeatures.clear_asift_plans! — Method
clear_asift_plans!()Drop all cached AsiftPlans, releasing their device buffers immediately rather than leaving them to a finalizer.
The cache is keyed by (W, H, device, tilts, upsample, seed_sigma, nb_scales, nb_octaves) and holds at most set_max_cached_plans! plans (default 1), evicting the least recently used. Call this to release the resident plans outright — e.g. between sequences in a mixed-resolution corpus, or before handing the device to another workload. The freed memory is returned to the device (on CUDA, CUDA.jl's pool is reclaimed), so an allocator outside the backend's pool, such as ONNX Runtime's, can use it: a LoMaDescriptor or LoMaMatcher on ONNX Runtime's CUDA provider calls this before it runs.
VisualGeometryToolkitFeatures.set_max_cached_plans! — Method
set_max_cached_plans!(n) -> IntSet how many AsiftPlans stay resident (default 1). Evicted plans have their device buffers released immediately via VisualGeometryToolkitCore's free_device_buffers!.
VisualGeometryToolkitFeatures.AsiftPlan — Type
AsiftPlan(W, H; device = default_device(), tilts = :imas25, seed_sigma = 1.6,
input_blur = 0.5, nb_scales = 3, capacity = 2^20, chunk = 0,
upsample = 1, nb_octaves = sift_nb_octaves(W, H; upsample))The preallocated buffers of the ASIFT detector for W×H images on device (any KernelAbstractions backend): the tilted warps, the scale-space pyramids, the feature pool and the compiled kernels. asift_detect! runs the detector in them without allocating on the device.
asift_detect builds and caches plans itself (see set_max_cached_plans! and clear_asift_plans!), so most callers never construct one. Build one to own its lifetime, or to detect in a loop over same-size frames without the cache's lock.
tilts: the tilt schedule, atilt_schedulename (:imas25, the default, or:untilted, the identity alone: plain DoG-SIFT) or a vector of(t, φ)pairs (tilt and longitude in radians).seed_sigma,input_blur,nb_scales,upsample,nb_octaves: the SIFT pyramid's seed blur, the input's assumed blur, the scales per octaveS, how many times the first octave is the image upsampled 2× (a count,true/falseread as 1/0; the first octave is octave−upsample) and the octave count (by defaultsift_nb_octaves, whose coarsest octave is the same for everyupsample; one more at most). The extremum search reads the interior DoG slices2 … S + 1of every octave, so the searched scales run fromseed_sigma·2^(1.5/S)·2^(−upsample)toseed_sigma·2^((S + 0.5)/S)·2^(nb_octaves − 1 − upsample)input pixels: an upsampling more reaches an octave finer, an octave more one coarser (a 4240×2832 image: 1.13 … 230 px at the defaults, 0.57 … 230 px atupsample = 2, the top doubled by an octave more). Atupsample ≥ 2the input's blur (2^upsample·input_blurof the first octave's pixels) exceeds the defaultseed_sigma: the first level is then the upsampled image unblurred (VLFeat's and OpenCV's rule), and the first octave's finest levels hold more blur than their σ states.capacity: the initial size of the feature pool; it grows when a frame produces more extrema.chunk: how many warps stay resident at once;0sizes the chunk from the device's free memory.
A plan holds device memory (several GiB for a large image under the full schedule). It is mutable because its feature pool grows, and because plan.refine_dropped records how many extrema the last call dropped when their sub-pixel refinement failed (no unrefined detection is returned) or, with uncertainty = FeatureCovariance(), when their covariance was singular or NaN (no detection without a usable covariance is returned).
julia> using VisualGeometryToolkitFeatures
julia> plan = AsiftPlan(64, 48; tilts = :untilted);
julia> img = zeros(Float32, 64, 48); # (W, H): x along the first axis
julia> asift_detect!(plan, img)
ImasFeatureKA[]VisualGeometryToolkitFeatures._chunk_warps — Method
_chunk_warps(dev, nw, warp_bytes, chunk) -> IntWarps to keep resident per chunk. chunk > 0 is honoured verbatim (capped at nw); chunk <= 0 auto-sizes from device_free_bytes so the pyramids fit in _PYRAMID_BUDGET of what is free right now, less the room ONNX Runtime's CUDA sessions may take (_onnx_cuda_reserve).
VisualGeometryToolkitFeatures.asift_detect! — Method
asift_detect!(plan, img_xy; nms=:none, window_half=3, cutoff=3.0,
polarity=:none, band_octaves=0.0, band_reach=1,
intensity_thr=0.04, edge_thr=10.0,
boundary_K=3.0, uncertainty=nothing) -> Vector{ImasFeatureKA}Full batched IMAS-25 detection in the plan's preallocated buffers. With nms = :mxm or :ellipse the cross-warp NMS runs on-device and only the survivors are returned; :none returns the full boundary-filtered pool. The pool is sorted into a canonical order first (position, scale, response, then every other field), so the features and their order do not depend on the threads or the device's scheduling. polarity (:dark → response > 0, :light → response < 0) gates the pool BEFORE the NMS splat, so a blob of the other sign can never suppress a kept one. band_octaves > 0 restricts suppression to features of comparable scale: a feature in scale band b = ⌊log2(√det S) / band_octaves⌋ can only be suppressed by features in bands b ± band_reach (see _band_pool!); 0 is the whole-pool rule. The seed blur σ₀ — the pyramid's, and the one the reported σ is scaled by — is the plan's (AsiftPlan(W, H; seed_sigma)).
uncertainty = FeatureCovariance() also returns each feature's covariance: the result is then (; features, cov), cov[i] the FeatureCovariance covariance of features[i], a CovMat{3} over (x, y, log σ).
VisualGeometryToolkitFeatures.ImasFeatureKA — Type
ImasFeatureKAOne raw feature of the ASIFT detector, as asift_detect! and asift_detect(img; raw = true) return it, before conversion to a Detection by asift_detections. All fields are in the input image's frame:
x,y: the refined centre in 0-indexed OpenCV pixel coordinates (top-left pixel centre at (0, 0), 0.5 less than the COLMAP coordinates of aDetection);σ: the characteristic scale, in input pixels;intensity: the DoG response at the extremum, positive for a dark blob and negative for a light one;s_xx,s_xy,s_yy: the symmetric 2×2 shape factor of the elliptic frame, back-projected from the tilted warp it was found in;warp_idx: the index of that warp in the plan's tilt schedule (1 is the identity).
julia> using VisualGeometryToolkitFeatures
julia> f = ImasFeatureKA(10.0, 20.0, 2.0, -0.1, 2.0, 0.0, 2.0, 1);
julia> d = only(asift_detections([f]));
julia> d.center, is_light(d)
([10.5, 20.5], true)VisualGeometryToolkitFeatures.tilt_schedule — Method
tilt_schedule(name::Symbol) -> Vector{Tuple{Float64, Float64}}
tilt_schedule(tilts) -> Vector{Tuple{Float64, Float64}}The simulated camera tilts ASIFT detects under, as (t, φ) pairs (tilt and longitude in radians), by name or as given (tilts, any collection of pairs):
:imas25— IMAS-25 (the default ofasift_detectandAsiftPlan): the identity, 8 tilts of 2.62 and 16 of 5.18 at evenly spaced longitudes. Affine invariance for wide-baseline views.:untilted— the identity warp alone,[(1.0, 0.0)]: plain DoG-SIFT. Its detections localize best: on real panorama frames a tilted warp's detections land about 0.95 px rms from the untilted view's of the same structure, the untilted's about 0.3 px from debiased SIFT, so for views that differ by a rotation (a panorama) it is the better schedule.
Any other name is an ArgumentError.
VisualGeometryToolkitFeatures.FeatureCovariance — Type
FeatureCovariance()Ask asift_detect for each detection's 3×3 covariance over (x, y, log σ), by the implicit function theorem, for iid unit pixel noise.
asift_detect(img; uncertainty = FeatureCovariance()) returns detections whose frame is VisualGeometryToolkitCore's Uncertain(frame, Σ̃); the default uncertainty = nothing runs the detector unchanged. cov(d) is Σ̃, feature_frame(d) the frame, and d.center, d.σ and the metadata read as on any detection.
Model. A refined DoG detection is a stationary point of the detector's local quadratic model of the DoG in (slice, x, y). With Ĥ its finite-difference Hessian and C the covariance, under unit iid pixel noise, of that model's gradient at the detection, Σ̃ = J·Ĥ⁻¹·C·Ĥ⁻¹·Jᵀ: C is tabulated exactly for the discrete pyramid (warp, upsample, blurs, octave downsamples) once per plan, and J maps to the reported coordinates, then through the ASIFT back-projection.
Parameterization.Σ̃ is a CovMat{3} over (x, y, log σ) in the detection's image frame (COLMAP pixel centres, as d.center), σ = d.σ (the frame's characteristic scale √det S), log natural. Position-only consumers take the top-left 2×2 block (the marginal). A tilted ASIFT warp's frame has the shape S = σ_w·M_w with M_w fixed by the warp (d.warp_idx into the tilt schedule), so (x, y, log σ) are its only free parameters and Σ̃ is full rank.
Units. Unit noise variance in the detector's intensity units: those of a float input image, and the [0, 1] scale for 8-bit input (UInt8, N0f8, …; σn = σgray / 255). The physical covariance is σ_n²·Σ̃.
Every detection is refined, and has a covariance: the detector searches the interior scale slices only and drops an extremum whose sub-pixel refinement fails, and with FeatureCovariance it drops, exactly so, a detection whose covariance is singular or NaN (the implicit-function system singular: not finite and positive definite), before the cross-warp NMS; plan.refine_dropped counts both. Every returned detection's Σ̃ is finite and positive definite.
Extension point (not implemented): an intensity-dependent noise model, e.g. FeatureCovariance(noise = ShotNoise(a, b)), makes the middle term K·diag(σ²(I))·Kᵀ, gathered from the pyramid per detection instead of tabulated.
VisualGeometryToolkitFeatures.GaussianScaleSpace — Type
GaussianScaleSpaceThe octaves' Gaussian levels of an image, built by gaussian_scale_space (one slot), and held by every AsiftPlan (its space, one slot per resident tilted warp: ASIFT's DoG pyramid is the difference of these levels).
levels[o]: octaveo'sS + 3levels, a(W_o, H_o, S + 3, slots)array on the device,(x, y)order (x along the first axis, as ASIFT holds images); levell(1-based) has the blurseed_sigma·2^((l − 1)/S)in the octave's pixels — or more, in a first octave upsampled more than once whose input blur2^upsample·input_blurexceedsseed_sigma(its first level is then the upsampled image itself, seeAsiftPlan). LevelS + 1of an octave is the next octave's level 1 before its 2× (Lowe's pixel-aligned) downsample.dims[o]:(W_o, H_o);first_octave:−upsample, the count of 2× upsamplings of the first octave (−1the 2× upsampled image,0the image itself; octaveohas pixels of2^(first_octave + o − 1)input pixels);nb_scales(S),seed_sigma, the input's assumed blurinput_blur(input pixels), and the per-level separable blurtaps(one-sided,gaussian_taps) and theirradii.
level_sigma gives a level's blur in input pixels.
VisualGeometryToolkitFeatures.gaussian_scale_space — Method
gaussian_scale_space(img; device = default_device(), nb_scales = 3, seed_sigma = 1.6,
input_blur = 0.5, upsample = 1, nb_octaves = sift_nb_octaves(W, H; upsample))
-> GaussianScaleSpaceThe Gaussian scale space of the grey image img (H×W, read as asift_detect reads it) on device: nb_octaves octaves of nb_scales + 3 levels each, by ASIFT's own pyramid (the same taps, the same fused tiled blur kernel, Lowe's pixel-aligned 2× downsample and the package's 2× upsampling of the first octave, upsample times), level for level what asift_detect builds for the untilted warp; upsample, nb_octaves and the seed σ as for AsiftPlan. See GaussianScaleSpace.
VisualGeometryToolkitFeatures.gaussian_taps — Method
gaussian_taps(sep_sigma) -> Vector{Float32}One-sided normalized Gaussian taps [k0, k1, …, kr] (center + r taps), with r = max(1, ceil(4·sep_sigma)). Normalized so k0 + 2·Σ kᵢ = 1.
VisualGeometryToolkitFeatures.level_sigma — Method
level_sigma(space::GaussianScaleSpace, o, l) -> Float64The Gaussian blur of level l (1-based, fractional allowed) of octave o (1-based) of space, in INPUT pixels: seed_sigma·2^((l − 1)/S)·2^(first_octave + o − 1) (the nominal blur: see GaussianScaleSpace for a first octave upsampled more than once).
VisualGeometryToolkitFeatures.sift_nb_octaves — Method
sift_nb_octaves(W, H; upsample=0) -> IntAuto octave count: floor(log2(min(W,H))) − 4, plus one per 2× upsampling (upsample, a count; true/false are 1/0), so the coarsest octave is the same whatever the upsampling: its shorter side is at least 32 px. One octave more, a shorter side of at least 16 px (VLFeat's count), is the most a pyramid holds (nb_octaves of gaussian_scale_space and AsiftPlan).
VisualGeometryToolkitFeatures.sift_sep_sigmas — Method
sift_sep_sigmas(seed_sigma, input_blur, nb_scales) -> Vector{Float32}Per-level separable-blur σ for one octave (nbscales+3 levels). Level 0 lifts the assumed `inputblurup toseedsigma`; each subsequent level advances by a factor 2^(1/nbscales) in absolute scale.
VisualGeometryToolkitFeatures.imas_rotated_canvas — Method
Rotated-canvas geometry for a θ rotation of an nx×ny image (Float32, matches C).
VisualGeometryToolkitFeatures.cross_warp_nms — Method
cross_warp_nms(feats, W, H; device, mode=:mxm, window_half=3, cutoff=3.0,
band_octaves=0.0, band_reach=1) -> Vector{ImasFeatureKA}Cross-warp NMS over an aggregated feature pool (input-frame). mode is :mxm (centroid window window_half) or :ellipse (Mahalanobis footprint, cutoff cutoff). band_octaves > 0 restricts suppression to features of comparable scale — band ⌊log2(√det S) / band_octaves⌋ ± band_reach. It runs the kernels asift_detect! runs on its device pool, on the pool in the same canonical order (exact ties in the quantized response break toward the later feature in that order), so the survivors do not depend on the order of feats, and on the pool asift_detect(img; nms = :none, raw = true) returns it keeps exactly the features asift_detect(img; nms = mode, raw = true) keeps, given the same detection keywords. Returns the survivors in that canonical order.
VisualGeometryToolkitFeatures.detect_sfop — Method
detect_sfop(img; device = default_device(), noise_sigma = 2.0, significance_level = 0.999,
precision_threshold = 0.5, nb_scales = 3, input_blur = 0.5, upsample = 1,
nb_octaves = min(3, sift_nb_octaves(size(img)...; upsample)),
seed_sigma = 2^upsample * input_blur, polarity = :none) -> StructArray{<:Detection}
detect_sfop(L::LinearRaw; sensor_model::SensorModel = SensorModel(), kwargs...)SFOP scale selection (Förstner et al., 2009) for CIRCULAR points (the spiral model at α = 90°): each point where the precision w of a circular point's estimate peaks in position and scale, with the integration scale σ it peaks at. Rows Detection(Disc(x, σ), (; precision)), COLMAP pixel centres of the input (of a RAW's green cells, L::LinearRaw: the cell grid, their values) and σ in input pixels; precision is w (px⁻²). Seed detect_foerstner(img, seeds) with them to localise each point at full resolution in its own window.
- The scale space:
gaussian_scale_space(img; nb_octaves, nb_scales, seed_sigma, input_blur, upsample), ASIFT's pyramid, 3 octaves by default or as many as the image holds (gaussian_scale_space'ssift_nb_octaves) when that is fewer (anb_octavesbeyond the one more a pyramid holds is refused).upsamplecounts the first octave's 2× upsamplings (true/falseread as 1/0; the defaultseed_sigma,2^upsample·input_blur, fits the pyramid's blur taps atnb_scales = 3up toupsample = 1). Its first level is the input itself (seed_sigmathe input's own blur in the first octave's pixels: no seed blur), so the finest scale searched is the finest the image holds: with the defaults the integration scales run from3·2^(2/3)·0.5 ≈ 2.4input pixels (the second level of the 2× upsampled octave) up. Levellof octaveo(blurσₗ, octave pixels) gives the differentiation scaleτ = σₗ₊₁, the next level's blur:gaussian_gradienton the level at the pyramid's own blur increment√(σₗ₊₁² − σₗ²), so every octave's levels sit on one ladder of scales and an octave's last levels are the next one's first; the integration scale isσ = 3τ. - The precision:
w = (N(σ) − 2)·λ₂(M̄)/Ω̄,N(σ) = 12σ² + 1(σ in the octave's pixels, sowis in px⁻² of the input at every octave),M̄the structure tensor (the squared gradient's average under the unit-sum GaussianG_σ),Ω̄ = Σ G_σ(d)·(dᵀ ∇g⊥)²the circular model's weighted squared distances about the window centre (eq. 5 of the paper at α = 90°). - Maxima: of
wover the 26 neighbours in (x, y, level), levels 2 … S + 2 of every octave (adjacent octaves share a level), refined by the quadratic interpolation of ASIFT's extraction on−1/w(sub-pixel, sub-level; the reference point's misfit is quadratic in its offset), or kept on its pixel and level where that interpolation fails (where clutter enters the windowwsteps);precisioniswat the sample. - Significance:
λ₂ > T,T = ½E[tr M̄]·χ²_ν(significance_level)/νfor white noise ofnoise_sigmagrey levels of an 8-bit image:E[tr M̄]and the degrees of freedomν = 2E²/Var(Satterthwaite) of the Gaussian-weighted sum of the correlated squared gradients, both from the gradient field's exact covariance under the level's blur. No empirical factor: the paper's 1.5 was the correlation the independent-pixel count2N(σ)ignored.λ₂ ≤ tr/2, so the test is conservative. - Precision: a maximum is kept when its point standard deviation
1/√wis at mostprecision_threshold(input pixels,detect_foerstner's keyword and default): the paper'sT_w. - Accuracy: seeded
detect_foerstnerat the selected σ equals the best fixed window within Monte Carlo error at dot radii 0.9–5 px, except near 1.8 px, where it is 0.0008 px RMS worse (0.0092 vs 0.0084 px): a measured, accepted limit of the window estimator's pixel locking (no window rule tested removed it). - Suppression: ASIFT's elliptical NMS (
cross_warp_nms), the paper's eq. (8) with its defaults (fp = 1, fs = ½, T_r = 1) as footprint overlap: a footprint of radiusσ/√2and half-octave scale bands, ± 1 band. No further deduplication: one blob found at two scales more than a band apart (the NMS keeps both) gives two rows, which the consumer resolves by its own geometry. - Polarity (
polarity = :none | :dark | :light, ASIFT's vocabulary and sign)::darkkeeps the dark blobs,:lightthe light ones, by the sign of the scale-normalised Laplacian of Gaussianσ²∇²(G_σ ⋆ I)at each candidate's selected σ (positive at a dark blob, ASIFT's DoG sign;is_dark), tested one-sided atsignificance_levelunder the λ₂ test's noise: a candidate whose sign the noise could have given (a window on flat background between dots) is kept for neither. Applied before the NMS (ASIFT's order: a blob of the other sign never suppresses a kept one).:none(the default) applies no gate: the structure tensor is sign-invariant and bright and dark blobs alike are found. - Noise:
noise_sigma(grey levels of an 8-bit image) is one level for the whole image. On a RAW's green cells under a gain-carryingsensor_model(the frame's, the oneLwas linearised with, soLcarries each value's variance) the λ₂ and polarity tests read each pixel's variance instead (noise_sigmaunread): the floor is linear in the noise variance, and each window's is the variance averaged underG_σ. Shot noise in a bright region then raises no blob. Without a gain,noise_sigmaas for a matrix.
VisualGeometryToolkitFeatures.detect_foerstner — Method
detect_foerstner([T = Float32], m::RawMosaic, seeds; sensor_model::SensorModel = SensorModel(), device = default_device(),
kwargs...) -> StructArray{<:Detection}The seeded Förstner window estimation on a RAW mosaic's bilinear green image on the raw grid (the image of detect_sfop(m), its seeds; the detector's own 2× doubling, ipfop.m step 1), in raw COLMAP pixels. Under a gain-carrying `sensormodelthe covariance is propagated from the PHOTOSITES: each green photositekmoves the estimate byJk = Σq J[:, q] W[q, k],Wthe bilinear weights (1/nat each pixel whosenaveraged greens include it: its own site, its red and blue 4-neighbours, and a defective green's 3×3 fallback), a censored (saturated) photosite with the law's variance at the white level, andΣ = Σk σk² Jk Jkᵀ,σk²the variance law's; without a gain the a-posteriori covariance. The F-test's effective degrees of freedom are those of the residual sums under that photosite noise (white over the green photosites, through the interpolation), not white input pixels. A window reading a pixel without a value (seedetectsfop(m)) is an:unknownrow. The other keywords and the rows are those of [detectfoerstner(img, seeds)`](@ref detectfoerstner).
VisualGeometryToolkitFeatures.detect_sfop — Method
detect_sfop(m::RawMosaic; sensor_model::SensorModel = SensorModel(), device = default_device(), kwargs...)SFOP on a RAW mosaic's green photosites at the raw resolution: the scale space of the BILINEAR green image on the raw grid (a green photosite its own value, a red or blue one the mean of its usable green 4-neighbours). A saturated photosite is censored: a normal sample at the white level (1) with the noise law's variance there. A defect of the sensor model is missing: it drops out, and a pixel with no usable green under its cross (a defective green) is the mean of the usable greens of its 3×3 neighbourhood; only a pixel with none in the 3×3 (a defect cluster) has no value, and every window reading it fails the λ₂ test. With SFOP's 2× upsampled first octave and input_blur the image's stated blur, 1/√3 raw pixels (the photosite's box ⊕ the interpolation). Rows as for a matrix, in raw COLMAP pixels (σ in raw pixels): the seeds of detect_foerstner(m, seeds).
Under a gain-carrying sensor_model (a Poisson–Gaussian, the frame's) the λ₂ and polarity tests read the image's noise from the photosites' variance law (each pixel's white-equivalent variance: twice the green photosites' under the cross, one green photosite per two raw pixels); without a gain noise_sigma as for a matrix. The greens G1 and G2 are taken to respond alike (the sensor model carries no per-channel response): a G1/G2 mismatch remains as a checkerboard of that amplitude (a known limit). input_blur and upsample are the image's and are refused; the other keywords are detect_sfop's.
VisualGeometryToolkitFeatures.spread_strongest — Method
spread_strongest(detections, image_size, n; tolerance = 0.1, device = default_device()) -> detectionsAbout n of the strongest detections spread over the image: Suppression via Square Covering (SSC; Bailo et al., Pattern Recognition Letters 106, 2018, ported from the authors' code, https://github.com/BAILOOL/ANMS-Codes, MIT). In descending |response| (ties: the earlier detection first), a detection is kept unless a kept one lies in the square of side w about it (on a grid of cells of side w / 2); the side w is an integer number of pixels, found by the authors' binary search until the kept count is within round(n ∓ n·tolerance). The kept detections are returned in detector order.
Where the global top n by |response| would sit (on a frame with a bright window and a dim interior) nearly all in the window, this keeps the strongest of each neighbourhood, at whatever contrast the neighbourhood has.
image_size is (H, W) or a channelview's (C, H, W); the detections' centroids are in its COLMAP pixel frame ([0, W] × [0, H]). With n or fewer detections, detections itself is returned; n = 1 is the single strongest. Where no integer width lands in the band (the reference then returns its last covering, outside the band), the width is refined between the two bracketing integers by bisection, and if the count still jumps over the band, the covering above it loses its weakest detections down to the band's top; whenever the reference's search lands in the band, the result is the reference's. The count is below the band only when every covering is (fewer detections than the band can hold).
One KernelAbstractions implementation on the CPU or any GPU backend (device): the covering is computed in parallel (a per-cell maximum, then synchronous rounds of the greedy independent set of the cells in score order), and its result is the reference's sequential sweep's exactly, ties included; every backend returns the same set, run to run.
VisualGeometryToolkitFeatures.detect_image_circle — Method
detect_image_circle([rng,] [T = Ellipse,] frames; outlier_halfwidth, initial = nothing, rim_band = (0.8, 0.0),
rim_angles = :auto, significance = 0.01,
device = default_device())
-> (; circle::T, fit, contour, cut, rim_gap, covariance, center_scale, axes_scale, rim::RimProfile) | nothingThe image circle of a lens MEASURED from photographs frames — an iterable of images of one size (a vector, or a generator that streams them: one camera, one lens, one focus, so the circle is the same in all of them) — in the frames' COLMAP pixel frame: the lens COVERAGE, where its light ends. No lens model predicts it (a model's fold says where the MODEL is valid, not where the field stop or the lens's own image circle cuts the light). The lit region is not necessarily round: it is fitted as the model type T, Core's Ellipse (the default) or Circle.
The fit is VisualGeometryToolkitEstimation's find_ellipse / find_circle, through this package's extension on it: using VisualGeometryToolkitEstimation first (without it the call is an ArgumentError naming the package).
The frames are gray images (anything Colors.gray reads: Gray, a real) or camera RAW images, never both:
- a
LinearRawof mode:green_cellsor aRawMosaic(read asLinearRaw(mosaic; mode = :green_cells)) is measured on its GREEN CELLS, the geometry image, its saturated cells excluded, and the result is taken to the raw's pixels by the cells'raw_frame_map(Core's similarity action on anEllipseor aCircle). The keywords in pixels are in GREEN-CELL pixels, the resolution measured on (half the raw's). Every frame must have the same map (one sensor, one window), and a cell image on a device is read to the host. Cells read under a sensor model with a variance law (anUncertainimage) carry each value's variance, and the rim's noise is then the MODEL's (SensorModelNoise); otherwise it is the data's (EmpiricalNoise). The type decides. A known issue (CHANGELOG 0.9.5): under a sensor model the faint light beyond the rim (a lens barrel's reflections, a halo) can be taken as the rim, its drops being far above the model's noise alone; the default path (no sensor model) is unaffected: measure the coverage without one until this is fixed; - any other image is measured as it is (its white, 1, is its saturation), and its result is in its own pixels.
Each frame lights a different part of the rim, and a scene's own edges move between frames; their union of light does the measuring:
- the per-pixel MAXIMUM over the frames (a region dark in one frame is filled by another; a scene edge is washed out), saturated where any frame is;
- resampled by VisualGeometryToolkitCore's log-polar warp (
logpolar_map,image_warp!) about the initial model's centre —initial, by default the image centre with the half diagonal for radius — over the radii(1 − rim_band[1])·min(axes)…(1 + rim_band[2])·max(axes)(the default: 0.2…1 of the half diagonal, every rim inside the frame down to a fifth of it), at most 1 px apart in radius (Δ(ln r) ≤ 1/r_max) and in arc (Δθ ≤ 1/r_max) at the outer end, so every pixel of the band contributes. The sample rows are grouped into angle bins and every (angle bin, radius) reduced to its MEDIAN (one KernelAbstractions kernel), both ondevice. A bin is skipped (NaN,:off_frameinrim.bin_reason) when a sample's bilinear taps leave the frame. A bin more than half of whose samples are saturated is FLAGGED, not skipped::saturatedinrim.bin_reason, its median kept in the profile and read like any other. The angle bins (rim_angles = :auto) are chosen from the image: of the binnings doubling from 45 (8°) to 3 sample rows per bin, the one whose coverage is the most precise (the smallest standard error of its mean radius, the inliers' rms over √(wedges with an inlier), the rms no finer than the edges' own median uncertainty nor the radial grid's spacing over √12, with at least 20 such wedges). When no binning has a fit with 20 such wedges, the finest binning tried is the result (its reasons recorded; without a fit,nothingis returned). Finer bins give more wedges but noisier medians and, in a textured scene, the scene's edges among the candidates; - per angle, the CANDIDATES: every EDGE of the profile's OUTER ENVELOPE — the brightest it gets at or beyond each radius: where the light ends it does not come back, while a scene edge or a dark object inside the disc is followed by light again and leaves the envelope alone. Every radius is in pixels through the log-polar map, its differential and its inverse (Core's
transform_deriv,inv): the drop at a radius r is across exactly 2 px about it (the bilinear taps' support; its ends r ± 1 px through the inverse map, the profile interpolated between columns), so it is continuous in r. An edge is a maximal run of significant drops; its candidate is the steepest drop in LOG brightness per pixel from the run's start to the next edge's (the log floored at the angle's darkest level or its noise), at its sub-sample peak in the grid coordinate taken to pixels by the map. A log drop is a ratio, independent of the exposure: a blurred sharp stop's steepest log drop is at the stop, a soft ramp's at its foot, where it meets the floor; a rim that falls in two steps (lit, a dim band, the floor) gives both (owner ruling 2026-11-14). A drop is significant when it exceeds its noise √(σ(r − 1)² + σ(r + 1)²) (σ each bin's noise at its pixels, rows × arc from the map's differential: the sensor model's, or the data's) by z = Φ⁻¹(1 − α/m),α = significancethe family-wise level over the angle's m drops: the floor's noise alone makes an edge in an angle with probability at most α. A noise of zero (a noise-free image: the data's noise is 0) is used as it is, with no floor: every positive drop is then significant, each edge's uncertainty is 0 (so the:autorule's floor on the rms is 0; its grid term remains), and the log is floored at the angle's darkest level (at leastfloatmin); - the model:
find_ellipse(candidates; scoring, rng)forEllipse,find_circle(…)forCircle— RANSAC with the robustified-LM local optimization, scored by the Student-t + uniform marginal (the inlier scale marginalized; the outliers uniform over ±outlier_halfwidth; its priors the stack's default, VisualGeometryToolkitEstimation'sscoring_from_tomlofstudent_t_uniform), on ONE candidate per angle, so the support counts angles and an angle with many edges does not dominate: first each angle's OUTERMOST edge (where its light ends); then each angle's edge NEAREST that model, any of its edges (stray light, or a step off the model, gives way), fitted again: the coverage. Per angle that edge, an inlier, is the rim's edge there (rim.edge); an angle whose rim is hidden (an object against the lens) is an outlier.
Returns nothing when no model is found (fewer than 5 candidates, or no consensus: a lens that lights the whole frame has no rim) or the model contains the whole frame (its light covers the sensor). Otherwise:
circle: the lens coverage, anEllipse{Float64}or aCircle{Float64}(T), COLMAP pixels (a RAW frame's: raw pixels);fit: the second RANSAC'sRansacEstimate, on one candidate per angle that has any (each angle's edge nearest the first model, in angle order);inlier_mask(fit)indexes those points. For RAW frames the RANSAC ran on the green cells: its estimate is mapped to raw pixels (Core'smap_estimate,estimate(fit) == circle), but itsrmsand every residual or scale in its metadata stay in GREEN-CELL pixels (raw pixels ÷ 2);contour: the rim's points,Vector{Point2{Float64}}, incircle's pixels: per angle bin with one, the inlier candidate nearest the coverage (rim.edge);cut: whether the coverage reaches past the frame (an APS-C lens on the full frame: the rim is seen in arcs);rim_gap: the largest angle, radians, between consecutive inliers around the coverage's centre. Below π the rim is seen on all sides of the centre and the centre and radius are measured; above π the observed rim is one arc, and the boundary away from it is the model's extrapolation;covariance: the coverage's covariance in Core'sambient_coordinatesof the MEASURED plane (an ellipse's centre and form(Q₁₁, Q₁₂, Q₂₂); a circle's centre and radius):s²·solver_covarianceon the inliers,s²their squared distances to the model along their rays over n − dof (nothingwhen singular, and for aCircle: its problem provides no solver covariance);center_scale,axes_scalethe standard deviations of the centre and of the semi-axes(a, b)incircle's pixels (NaNwithout a covariance);rim: the measurement (RimProfile): the profiles, every angle's candidates and edge, the noise statistics.
Keywords
outlier_halfwidth(required, pixels > 0): the Student-t + uniform scorer's outlier half-width; a literal of the caller's, no computed default. 1000 full-resolution px held one basin on every set measured, for 750–2000.initial: the initial model (the result's pixels): its centre is the log-polar origin, its axes scale the band. By default the image centre with the half diagonal (no prior: a camera's principal point is a better centre when known).rim_band=(inner, outer), fractions of the initial model's axes: the radii resampled, from(1 − inner)·min(axes)to(1 + outer)·max(axes). The default(0.8, 0.0), with the default initial (the half diagonal), resamples from a fifth of the half diagonal out to the corners. The outer end is derived: the corners are the farthest any light reaches on the sensor. The inner end is a stated ASSUMPTION, not a fit: the coverage reaches at least a fifth of the half diagonal (a lens whose light ends nearer the centre leaves 96 % of the sensor dark, no use as a camera). It cannot be 0 because the log grid needs a positive inner radius, and the band's cost grows only as ln(1/0.2)·r_max radial samples. Narrow it when the coverage is known roughly, e.g. from a previous run.rim_angles: the angle bins,:auto(chosen as in step 2) or a count ≥ 5.significance: the edge test's family-wise level per angle: the probability that the floor's noise alone makes an edge anywhere in an angle. 0.01 is the conventional 1 % level, stated, not tuned. A false edge is one more candidate, and it is not consistent with the coverage, so the RANSAC rejects it as an outlier; the level only trades such outliers against missing a faint rim's edge.device: the KernelAbstractions backend of the warp and the bin medians (default_device(): the GPU when one is functional).min_hole_area,smoothing_weight,border_margin: the hull fit's keywords before 0.9.2, accepted and ignored with a deprecation warning (the hull is gone).
Examples
using VisualGeometryToolkitFeatures, VisualGeometryToolkitEstimation
using Random: Xoshiro
res = detect_image_circle(Xoshiro(1), frames; outlier_halfwidth = 1000)
res === nothing || (res.circle, res.axes_scale) # the coverage, an Ellipse, the frames' pixels, and its σ
raws = (load_raw(f; mode = :green_cells) for f in files)
detect_image_circle(Xoshiro(1), Circle, raws; outlier_halfwidth = 500).circle # keywords in cell px, circle in raw pxVisualGeometryToolkitFeatures.EmpiricalNoise — Type
EmpiricalNoise()A RimProfile's noise statistics (its statistics) when the frames carry no variances: each angle's noise is the data's own, Core's mad_scale of the profile's drops across 2 px, each over its noise per unit pixel noise (the bins' pixels).
VisualGeometryToolkitFeatures.RimProfile — Type
RimProfileThe measurement behind detect_image_circle's coverage (its rim field): the log-polar profiles of the frames' per-pixel maximum around the initial model, every angle's candidates and its chosen edge.
Fields (radii and centre in the MEASURED plane's pixels: a RAW frame's green cells):
center::Point2{Float64},initial— the log-polar origin and the initial model it is the centre of (the measured plane's pixels);angles::Vector{Float64}— each angle bin's centre, radians,atan(y − cy, x − cx)(COLMAP pixels, y down);radii::Vector{Float64}— the samples' radii fromcenter, ascending,Δ(ln r) ≤ 1 / r_max(≤ 1 px at r_max);bin_samples::Vector{Int}— each angle bin's sample rows (≤ 1 px of arc apart at r_max);profile::Matrix{Float64}—profile[a, j], the median of angle binaat radiusj,NaNwhere the bin is skipped (:off_frame); a:saturatedbin is flagged, not skipped: its median is kept and read like any other;bin_reason::Matrix{Symbol}—:ok,:off_frame(a sample's bilinear taps leave the frame) or:saturated(more than half the bin's samples saturated: a flag, the median kept);statistics—SensorModelNoise()orEmpiricalNoise(), chosen by the frames' type (the provenance);profile_scale::Matrix{Float64}each bin's noise under the sensor model (NaNwithout one): the median over the bin's meff distinct pixels of the maximum of the N frames' noise, standard error `√v̄ / (2·N·φ(q)·Φ(q)^(N−1)·√meff),q = Φ⁻¹(2^(−1/N)), v̄ the model's variance there (one frame:√(π/2·v̄/meff)`); meff the bin's rows × its cells' arc, from the log-polar map's differential (Core'stransform_deriv): its pixels (a column is at most a pixel thick, sharing its pixels with its neighbours);noise::Vector{Float64}— each angle's noise PER PIXEL σ₁ (a bin of meff pixels has σ₁/√meff): the model's (the median over the angle of its bin noise × √meff) or the data's; `modelscatter::Vector{Float64}— with a sensor model, the empirical noise over the model's per angle (≈ 1 when the model holds;NaN` without one);candidates::Vector{Vector{Float64}}— per angle the radii of its candidates;edge,edge_scale::Vector{Float64}— per angle the chosen candidate (NaNwhere none is an inlier of the coverage) and its standard uncertainty, the drop's noise through the profile's slope;reason::Vector{Symbol}— per angle:ok,:no_candidate(no significant drop) or:no_inlier.
VisualGeometryToolkitFeatures.SensorModelNoise — Type
SensorModelNoise()A RimProfile's noise statistics (its statistics) when the frames are Uncertain cells (a RAW read under a sensor model with a variance law): each bin's noise is the model's.
The opt-in continuous refinement (asift_detect(img; refinement = ContinuousRefinement())):
VisualGeometryToolkitFeatures.ContinuousRefinement — Type
ContinuousRefinement(; prefilter_halfwidth, spatial_bound, scale_bound, min_curvature)The continuous DoG-extremum refinement, opt-in: asift_detect(img; refinement = ContinuousRefinement()) (the detector's default, refinement = nothing, is today's one-step refinement). One recipe: the spatial gradient is Farid and Simoncelli's matched first-derivative filters on the prefiltered slices (IEEE TIP 2004), interpolated by the cardinal cubic spline; the scale row is the cardinal spline of the UNFILTERED slices, Lagrange in scale — so the prefilter sharpens the position and leaves the scale unbiased; the LM's Jacobian is the exact derivative of that gradient. The keywords are the bounds of _refine_dog_extremum; every default is configs/continuousrefinement.toml's (its comments give each derivation). The LM's stop settings are VisualGeometryToolkitCore's, the stack's one owner (StaticLmStops, its [lm] table); `maxiter,convergedtol2andcostfloor` are retired here. An isbits value, so a kernel takes it.