Matching
Strategies, metrics and matches; the matchers on the CPU, on a GPU and over stacked galleries; described features; the LoMa networks.
VisualGeometryToolkitFeatures.nn_match — Function
nn_match(desc1, desc2, strategy = MutualNN(); device = default_device(), kw...) -> Vector{NNMatch}match on device, any KernelAbstractions backend (VisualGeometryToolkitCore's default_device() when none is supplied: an accelerator when one is functional, else KA.CPU()). Every backend runs the same GEMM and selection kernels, so a GPU (e.g. CUDABackend()) returns what the CPU returns, with every keyword match accepts: metric, dm, ratio_threshold, min_similarity, max_distance.
VisualGeometryToolkitFeatures.CorrespondenceSet — Type
CorrespondenceSet(features::AbstractVector)
CorrespondenceSet(features::Tuple)
CorrespondenceSet(T::Type)An N-way correspondence: the same scene point observed as one feature in each of several images, where Correspondence is the two-way case. It wraps a vector of features and iterates, indexes and has a length like one. A tuple becomes a fixed-size SVector; a Vector (or CorrespondenceSet(T), an empty one of element type T) grows with push! and append!.
julia> using VisualGeometryToolkitFeatures
julia> track = CorrespondenceSet(Int);
julia> push!(track, 3); push!(track, 7);
julia> length(track), track[2], collect(track)
(2, 7, [3, 7])
julia> length(CorrespondenceSet((1, 2, 3)))
3VisualGeometryToolkitFeatures.CosineSimilarity — Type
CosineSimilarity <: DescriptorSimilarityCosine similarity for L2-NORMALIZED descriptors, computed as the dot product a·b — no normalization is applied, so un-normalized input gives raw dot products. Range [-1, 1] for unit vectors, 1 = identical.
VisualGeometryToolkitFeatures.DescriptorDistance — Type
DescriptorDistance <: DescriptorMetricDistance metrics: lower values = more similar. NN finds minima.
VisualGeometryToolkitFeatures.DescriptorMetric — Type
DescriptorMetricAbstract type for descriptor comparison metrics.
VisualGeometryToolkitFeatures.DescriptorSimilarity — Type
DescriptorSimilarity <: DescriptorMetricSimilarity metrics: higher values = more similar. NN finds maxima.
VisualGeometryToolkitFeatures.KnnMatch — Type
KnnMatch(k) <: NNStrategyk-nearest-neighbor matching: for each desc1 row, keep its k nearest desc2 columns instead of only the single best. Permissive by design — surfaces true matches that sit at rank 2..k or just under a strict cutoff (common with repetitive patterns, where a confuser can out-rank the true neighbor). Downstream geometry (e.g. RANSAC) disambiguates the extra candidates. Pair with min_similarity to bound the kept scores.
VisualGeometryToolkitFeatures.L2 — Type
L2 <: DescriptorDistanceEuclidean distance: ‖a - b‖. Range [0, 2] for unit vectors, 0 = identical.
VisualGeometryToolkitFeatures.MutualNN — Type
MutualNN <: NNStrategyMutual nearest-neighbor matching: find NN in both directions, keep only pairs where both directions agree.
VisualGeometryToolkitFeatures.NN — Type
NN <: NNStrategyOne-way nearest-neighbor matching: for each descriptor in desc1, find the nearest neighbor in desc2.
VisualGeometryToolkitFeatures.NNMatch — Type
NNMatchA nearest-neighbor match: Attributed{Correspondence{Int,Int}, NNMatchMeta}.
Property forwarding gives direct access:
m.first— index into desc1m.second— index into desc2m.score— match quality (similarity or distance, depending on metric)m.ratio— Lowe's SNN ratio
VisualGeometryToolkitFeatures.NNMatchMeta — Type
NNMatchMetaMetadata for a nearest-neighbor match.
Fields
score::Float32: Match quality in the metric's native units (similarity for CosineSimilarity, distance for L2)ratio::Float32: Lowe's ratio —d_best / d_2ndfor a distance metric,s_2nd / s_bestfor a similarity, so that smaller = more distinctive whatever the metric.Inf32when there is no second neighbour, and when a similarity's best score is non-positive: the ratio is undefined there, and no ratio test admits it.
VisualGeometryToolkitFeatures.NNStrategy — Type
NNStrategyAbstract type for nearest-neighbor matching strategies.
VisualGeometryToolkitCore.correspondences — Method
correspondences(features1, features2, matches::AbstractVector{NNMatch})Convert index-based NN matches to geometric correspondences.
Returns a columnar Correspondences whose endpoints are the matched features and whose attribute columns carry BOTH match statistics:
score::Vector{Float64}— higher-is-better sampling prior (similarity metrics are used as is; distance metrics are inverted1/(d + 1e-6))ratio::Vector{Float32}— the Lowe's-ratio value of each match (previously computed and then dropped; it now survives to downstream consumers)
Access endpoints via sources/targets, attributes via cs.score / cs.ratio; subsetting keeps all columns aligned.
VisualGeometryToolkitFeatures.match — Function
match(desc1, desc2, [strategy]; metric, dm, ratio_threshold,
min_similarity, max_distance) -> Vector{NNMatch}Match descriptors using nearest-neighbor search, on the CPU. It runs the same KernelAbstractions matcher as nn_match on a GPU, so a device returns what this returns. Descriptors are compared in their own float type.
Public but not exported, because it would shadow Base.match: import it (using VisualGeometryToolkitFeatures: match) or call VisualGeometryToolkitFeatures.match.
Arguments
desc1::AbstractMatrix{<:Real}: N1×D descriptor matrix (rows = descriptors)desc2::AbstractMatrix{<:Real}: N2×D descriptor matrixstrategy::NNStrategy:MutualNN()(default),NN()orKnnMatch(k)
Keyword Arguments
metric::DescriptorMetric:CosineSimilarity()(default) orL2()dm: Precomputed N1×N2 pairwise matrix (skippairwise_matrixcomputation)ratio_threshold: Lowe's SNN ratio threshold;nothing= disabledmin_similarity: Minimum score forDescriptorSimilaritymetricsmax_distance: Maximum score forDescriptorDistancemetrics
Returns
Vector{NNMatch} sorted by score (best first).
Usage
using VisualGeometryToolkitFeatures: match, pairwise_matrix, MutualNN, NN, CosineSimilarity, L2
# Cosine similarity (default) — good for L2-normalized descriptors
match(d1, d2, MutualNN())
match(d1, d2, MutualNN(); min_similarity=0.7)
# L2 distance
match(d1, d2, MutualNN(); metric=L2(), max_distance=0.8)
# Precomputed distance matrix
dm = pairwise_matrix(CosineSimilarity(), d1, d2)
match(d1, d2, MutualNN(); dm)
# With Lowe's ratio test
match(d1, d2, NN(); ratio_threshold=0.8)VisualGeometryToolkitFeatures.resolve_one_to_one! — Method
resolve_one_to_one!(used_a::AbstractVector{Bool}, used_b::AbstractVector{Bool},
candidates; a = c -> c.first, b = c -> c.second) -> Vector{Int}Greedy one-to-one reduction over an ALREADY-ORDERED candidate list: visit candidates in the given order, keep a candidate when neither of its endpoints is used yet, and mark both endpoints used. Returns the kept indices into candidates (in visit order); used_a/used_b are mutated in place, so pre-seeded entries (e.g. matches certified by an earlier stage) are respected and callers can chain several candidate lists through one pair of masks.
a/b extract the two endpoint indices from a candidate — the defaults serve NNMatch (m.first/m.second); supply accessors for tuple lists.
The caller owns the ORDER: sort best-first (descending similarity, ascending distance) for best-first assignment (the reduction MutualNN sets already satisfy, and the one KnnMatch(k) sets need), by overlap for geometric association, etc.
VisualGeometryToolkitFeatures.resolve_one_to_one — Method
resolve_one_to_one(matches::AbstractVector{NNMatch}, n_a, n_b;
metric = CosineSimilarity()) -> Vector{NNMatch}Best-first one-to-one reduction of an NNMatch list: visit the best match first and keep each first/second index at most once. metric must be the metric the matches were scored with, because it decides what "best" means: the highest score for a DescriptorSimilarity (the default, CosineSimilarity()), the lowest for a DescriptorDistance such as L2(). n_a/n_b bound the two index spaces. MutualNN output is returned unchanged (already one-to-one); KnnMatch(k) output collapses to its best consistent assignment.
VisualGeometryToolkitFeatures._nn_select — Method
_nn_select(strategy, metric, dev, offsets, sizes; gal, feat, S, chunk_bytes)
-> (top_idx, top_sc, rev_idx)The selection behind every match, on backend dev. The gallery is the rows, cut into blocks (offsets, 0-based, and sizes); the scores are either computed from descriptors already on dev (gal, rows × D, and feat, features × D, in the score type) a row range at a time, or read from a precomputed matrix S (rows × features). Returns host arrays: per row the top k feature indices and their signed scores (rows × k), and for MutualNN each feature's best row within each block (features × blocks, block-local), else nothing.
The score block is the pipeline's largest single allocation (M × N: 3.09 GiB for a 60k gallery against 13.7k detections, 9.79 GiB against 43.5k). Materialising it whole is what OOMs on cluttered frames, so it is built _row_chunks at a time and each block released eagerly via VisualGeometryToolkitCore's free_device_buffers! rather than waiting for a finalizer. The result does not depend on the chunking.
VisualGeometryToolkitFeatures._row_chunks — Method
_row_chunks(backend, M, N, T, chunk_bytes) -> Vector{Tuple{Int, Int}}Consecutive ROW ranges of an M-row gallery whose score block (rows × N of T) fits the budget. chunk_bytes > 0 is honoured verbatim; otherwise the budget is _SCORE_BUDGET of device_free_bytes, which returns typemax on backends exposing no query — yielding a single chunk, the historical behaviour.
Rows, not whole galleries: one calibration target with thousands of blobs is ONE gallery whose block alone can exceed the card (9.3 GiB against a 40k-feature frame, 2026-09-15), so a gallery may span several chunks. That is still exact — the forward sweep is per row, and the reverse sweep merges a gallery's per-column best across its chunks (_ka_top1_cols_blocked!).
VisualGeometryToolkitFeatures.pairwise_matrix — Function
pairwise_matrix(metric, desc1, desc2, device = default_device()) -> MatrixThe N1×N2 pairwise comparison matrix: a·b for CosineSimilarity(), ‖a − b‖ for L2(), in the descriptors' float type (Float32 descriptors give Float32, Float64 give Float64). The GEMM runs on device, any KernelAbstractions backend (CUDABackend(), …; VisualGeometryToolkitCore's default_device() when none is supplied: an accelerator when one is functional, else KA.CPU()); the matrix is returned on the host. One implementation serves every backend: there is no separate host method.
VisualGeometryToolkitFeatures.StackedDescriptorIndex — Type
StackedDescriptorIndex(backend, galleries::AbstractDict)Device-resident stack of descriptor galleries: all galleries' descriptor rows in one matrix, plus per-gallery offsets/sizes for the block-local kernels. galleries maps keys to descriptor matrices (rows = features) or to anything descriptor_matrix accepts. The stack keeps the galleries' float type (Float32 galleries stack as Float32, Float64 as Float64). Build once, reuse across nn_match_stacked calls.
VisualGeometryToolkitFeatures.nn_match_stacked — Method
nn_match_stacked(strategy, galleries::AbstractDict, features;
backend, index = nothing, min_similarity = 0.6, chunk_bytes = 0)
-> Dict{K, Vector{NNMatch}}Match features against every gallery in galleries with one stacked GEMM. Per gallery the result is identical to matching that gallery alone with strategy (MutualNN reverse checks are block-local; KnnMatch(k) takes any k): it is the matcher match runs, with the galleries as blocks of its rows. Matches use LOCAL indices: m.first indexes into gallery k's rows, m.second into features. Supply a prebuilt StackedDescriptorIndex as index to skip the per-call stack/upload.
When index is given it — not galleries — defines what gets matched: the returned dictionary has one entry per index.keys, and galleries supplies only the key type K. The two key sets therefore need NOT agree, and callers rely on that: a caller narrowing its gallery set across calls can keep ONE index built over the full set rather than restaging descriptors every call, and simply ignore the extra entries. Build the index from the widest set you will ever match against.
The score matrix is built a row range at a time. chunk_bytes > 0 caps each range's score block (rows × features) at that many bytes; the default 0 sizes the ranges from free device memory. The result does not depend on the chunking. Scores are cosine similarities in the wider of the index's and the features' float type.
VisualGeometryToolkitFeatures.DescribedFeature — Type
DescribedFeature{D<:Detection, N}A feature detection paired with a fixed-length descriptor vector.
Wraps a Detection (frame + metadata) with an SVector{N, Float32} descriptor. All detection properties (.center, .σ, .frame, .metadata, etc.) are forwarded transparently via @generated getproperty.
Use with StructArray for efficient columnar storage and zero-copy descriptor matrix extraction:
sa = described_features(detections, desc_matrix)
matches = VisualGeometryToolkitFeatures.match(sa_board, sa_query, MutualNN())
scored = correspondences(sa_board, sa_query, matches)VisualGeometryToolkitFeatures.described_features — Method
described_features(detections, desc_matrix) -> StructArray{DescribedFeature}Build a StructArray of DescribedFeature from a detection vector and an N×D descriptor matrix. The descriptor dimension D becomes the static size parameter of each SVector{D, Float32}.
VisualGeometryToolkitFeatures.descriptor_matrix — Method
descriptor_matrix(sa::StructArray{<:DescribedFeature}) -> AbstractMatrix{Float32}Zero-copy N×D descriptor matrix view from a StructArray of DescribedFeature. Suitable for supplying directly to match.
VisualGeometryToolkitFeatures.match — Function
match(f1::StructArray{<:DescribedFeature}, f2::StructArray{<:DescribedFeature},
strategy; kwargs...) -> Vector{NNMatch}Match described features by their descriptors. Extracts descriptor matrices and delegates to the matrix-based match.
VisualGeometryToolkitFeatures.LOMA_IMAGE_SIZE — Constant
LOMA_IMAGE_SIZE = 784The side of the DeDoDe descriptor graph's fixed square input: a LoMaDescriptor resamples an image of another size to (3, 784, 784).
VisualGeometryToolkitFeatures.LoMaDescriptor — Type
LoMaDescriptor(variant = :b; max_detections = 2048, bf16 = false, execution_provider = :auto,
threads = Threads.nthreads())A DeDoDe descriptor of LoMa: :b (DeDoDe-B, 128-dimensional, 54 MB) or :g (DeDoDe-G, 256-dimensional, 1.3 GB; 690 MB with bf16 = true, which needs ONNX Runtime's CUDA provider). The model is downloaded on first use and cached. On the CUDA provider its session's arena is capped and the detector's plans leave it room (they are sized without the memory ONNX Runtime may hold), so neither releases the other. (descriptor)(img, detections) describes about max_detections of the strongest detections of img, spread over it (spread_strongest at its default tolerance), and returns a StructArray{DescribedFeature}.
execution_provider is ONNX Runtime's: :auto runs on CUDA when onnx_cuda_status finds a device and the CUDA 12 libraries (the environment holds CUDA_Runtime_jll and CUDNN_jll; ONNX Runtime's CUDA build, conda-forge's, is installed on first use), and on the CPU with a warning naming why otherwise; :cuda is an error where CUDA is unavailable, naming why; :cpu is the CPU. ONNX Runtime's intra-op pool has threads threads (the inter-op pool 1; no core pinning). loma_provenance records the provider used and the threads.
The network is DeDoDe's (Edstedt et al., 3DV 2024), as exported by LoMa; cite both (BibTeX in the package README).
img is a (3, H, W) RGB channelview in [0, 1] of any size and real element type, as Float32.(channelview(rgb)) gives it, and detections are in its COLMAP pixel frame. The exported graph's input is fixed at LOMA_IMAGE_SIZE² and feeds the first convolution directly, with no normalization: an image of another size is resampled to it per channel by VisualGeometryToolkitCore's bilinear image_warp! (the COLMAP scaling x ↦ (W, H) / 784 ⊙ x, no prefilter), and a (3, 784, 784) image is taken as it is. The detection points enter the network's [-1, 1] chart from the image's own size, so the detections are never rescaled, and the described features carry them as given (frame in = frame out). Do not ImageNet-normalize the image: on LoMa's toronto pair that drops the matches from 166 to 14 (and none above probability 0.5).
A descriptor pairs with the LoMaMatchers of its dimension: :b with :b128, :g with :b, :l, :g and :r.
VisualGeometryToolkitFeatures.LoMaMatcher — Type
LoMaMatcher(variant = :b128; bf16 = false, execution_provider = :auto, threads = Threads.nthreads())A LoMa matcher: :b128 (48 MB) on 128-dimensional DeDoDe-B descriptors, or :b (48 MB), :l (184 MB), :g (726 MB) and :r (48 MB) on 256-dimensional DeDoDe-G ones (LoMaDescriptor(:g)). bf16 = true takes the half-size bf16 export of any of them, which needs ONNX Runtime's CUDA provider (its CPU provider lacks bf16 kernels). The model is downloaded on first use and cached; see the call method for matching. execution_provider (:auto, :cpu or :cuda) and threads are LoMaDescriptor's; loma_provenance records them.
The network is LoMa's (Nordström et al., "LoMa: Local Feature Matching Revisited", ECCV 2026); cite it (BibTeX in the package README).
VisualGeometryToolkitFeatures.LoMaMatcher — Method
(matcher::LoMaMatcher)(features_a, features_b, size_a, size_b; min_score = 0) -> Vector{NNMatch}Match two sets of LoMaDescriptor features, whose descriptor dimension must be the matcher's (ArgumentError otherwise). size_a/size_b are the image sizes, (H, W) or a channelview's (C, H, W).
Returns the same type as match: m.first indexes features_a, m.second indexes features_b, sorted best-first. m.score is the network's match probability (higher is better, a similarity, so the default correspondences(features_a, features_b, matches) reads it correctly); m.ratio is Inf32, since the network has no second neighbour. Matches scoring below min_score are dropped.
VisualGeometryToolkitFeatures.loma_provenance — Method
loma_provenance(l::Union{LoMaDescriptor, LoMaMatcher}) -> NamedTupleWhat runs l, as plain data for a result to record: model (the release file) and its sha256, the execution_provider actually used (:cpu or :cuda, after :auto is resolved), the ONNX Runtime intra-op threads, the onnxruntime version and, on CUDA, the cuda_driver, cuda_runtime and cudnn versions.
VisualGeometryToolkitFeatures.OnnxCudaStatus — Type
OnnxCudaStatusWhether ONNX Runtime's CUDA provider can run in this process, probed once: available, and reason (empty when available) naming what is missing. With the provider available, driver (the CUDA driver API version), cuda_runtime and cudnn (the versions of the libraries loaded for it) are set.
VisualGeometryToolkitFeatures.onnx_cuda_status — Method
onnx_cuda_status() -> OnnxCudaStatusONNX Runtime's CUDA provider in this process: probed (and its libraries loaded) on the first call, the same answer after. :auto runs on CUDA exactly when onnx_cuda_status().available.
VisualGeometryToolkitFeatures.onnx_run — Method
onnx_run(session, inputs) -> outputs
onnx_run(f, session) -> f()A run of an onnx_session the ASIFT detector's device budget counts: on a CUDA session the arena's growth is measured around the run (the drop in the device's free memory) and charged to that session's arena, so the detector's plans leave free only the room the arena may still take. onnx_run(session, inputs) runs session(inputs); onnx_run(f, session) runs f(), any call that runs session (a device-tensor ONNXRunTime.CAPI.Run on session.session, say), and returns its value. A CPU session's run is not measured. Every run of a package-built session goes through it; a run outside it leaves the arena's growth uncounted, and the detector's plans too large for the memory left.
VisualGeometryToolkitFeatures.onnx_session — Method
onnx_session(path; execution_provider = :auto, threads = Threads.nthreads(), gpu_mem_limit = 8 * 2^30)
-> ONNXRunTime.InferenceSessionAn ONNX Runtime session of the model file path, built as this package builds its own networks' (LoMa's): ONNXRunTime's load_inference with the session options it does not take. execution_provider is resolved by resolve_execution_provider (:auto runs on CUDA when onnx_cuda_status finds a device, on the CPU with a warning naming why otherwise; :cuda is an error where CUDA is unavailable; :cpu is the CPU), so a CUDA session runs ONNX Runtime's CUDA build this package installs, which has device code for every device from sm70 to sm120, the B200's sm100 among them (Microsoft's GPU builds lack sm100). The session's execution_provider field holds the resolved provider.
ONNX Runtime's intra-op pool has threads threads (its core pinning off), its inter-op pool one. On CUDA, the session's device arena is capped at gpu_mem_limit bytes and grows by what each run requests; a run that needs more is ONNX Runtime's out-of-memory error. The ASIFT detector's device plans leave free the room the process's live CUDA sessions may still take (each cap less what its arena holds, measured around the package's own runs; a session released or collected reserves nothing), so a session and the detector share the device without either releasing the other's memory. The plans cached before a CUDA session is created are released once, then. On the CPU gpu_mem_limit is ignored.
VisualGeometryToolkitFeatures.resolve_execution_provider — Function
resolve_execution_provider(execution_provider) -> SymbolThe ONNX Runtime provider execution_provider runs on here: :auto is :cuda when onnx_cuda_status finds the CUDA provider available, and :cpu with a warning naming why otherwise; :cuda is :cuda, or an error naming why it is unavailable; :cpu is :cpu. Anything else is an ArgumentError. What LoMaDescriptor and LoMaMatcher run on; a caller that keys a cache on the provider resolves it first.