Image Input
The image-input helpers, the linear-radiance PNG reader (load_linear_image) and the camera RAW reader (load_raw).
Camera RAW
The stack's one RAW path: every consumer reads RAW files through this module, and VisualGeometryToolkitCore's check_raw_decoding flags any other decoding path. load_raw(path; mode, gain) reads a Bayer RAW file in the sensor's orientation (the EXIF flip is never applied). The decoder is LibRawDecoder (using LibRaw_jll: LibRaw 0.22 through ccall, no Python). It produces VisualGeometryToolkitCore's RawMosaic, and every number after the decode is Core's: the mosaic's colour phase and its photosites channel-first (channelview), the linear values on the 2×2 cells (LinearRaw) and the exact maps onto the sensor (raw_frame_map, bayer_plane_map, green_cell_sites, raw_coordinates). Those names are exported by Core only (using VisualGeometryToolkitCore); their reference and design are Core's "Camera RAW" API page and "Raw images" guide, and this package's Raw images page says what is left here.
| mode | image | use |
|---|---|---|
:green_cells | each 2×2 Bayer cell's two greens, averaged; half resolution, one plane | geometry and radiometry |
:bin2 | the same cells with their red and blue (R, the green-cell plane, B) | colour radiometry |
:demosaic | LibRaw's AHD demosaic, full resolution, linear | display figures only |
The cell modes are exact: (raw − black[c]) / (white − black[c]) × gain, with saturation flags and each plane's true sample position, on the cell grid that maps onto the visible raw's PixelFrame{:colmap} by x ↦ 2x exactly. The demosaic interpolates. It has no frame map, so the geometric accessors refuse it. It takes a file or a mosaic in memory, load_raw(mosaic; mode = :demosaic): a mosaic corrected per photosite (vignetting, say) is displayed through the same LibRaw demosaic, on its photosites, phase and levels.
The EXIF fields become VisualGeometryToolkitCore's typed values through exposure_settings, lens_settings, body_identity, lens_identity and capture_time, from a mosaic or from load_raw_metadata (the headers only); extract_capture_settings gathers a capture's frames into a CaptureSettings. A lens's EXIF name is a claim: on an adapter (the FX3 with Sigma EF lenses) it names another lens, so a pipeline that knows its lens states it.
VisualGeometryToolkitFeatures.LibRawDecoder — Type
LibRawDecoder()Decode with LibRaw through LibRawjll (`using LibRawjll): accalllayer over LibRaw's C API, no Python. The extension checks LibRaw's version at load and the layout of every structure it reads against LibRaw's own accessors, and refuses (ErrorException`) a library it was not written for. The stack's one decoder.
VisualGeometryToolkitFeatures.RawDecoder — Type
RawDecoderA RAW decoder, the token load_raw and load_raw_mosaic dispatch on: LibRawDecoder, implemented by a package extension that needs its trigger package loaded.
VisualGeometryToolkitFeatures.body_identity — Method
body_identity(x) -> DeviceIdentityThe camera body a RAW file says took it (make, model, body_serial), as Core's DeviceIdentity; a field the file does not state is nothing.
VisualGeometryToolkitFeatures.capture_time — Method
capture_time(x) -> Union{String, Nothing}When a RAW frame was taken, as the file states it: EXIF's DateTimeOriginal (local time, no zone) with its SubSecTimeOriginal, written ISO 8601 ("2026-10-29T14:03:12.34"); nothing when the file states none.
VisualGeometryToolkitFeatures.exposure_settings — Method
exposure_settings(x) -> ExposureSettingsThe exposure a RAW frame was taken with — x a RawMosaic or its metadata (load_raw_metadata) — as Core's ExposureSettings: exposure_s seconds, fnumber, iso. A field the file does not state is an ArgumentError naming it.
VisualGeometryToolkitFeatures.extract_capture_settings — Method
extract_capture_settings(frames; lens = nothing) -> CaptureSettingsThe camera settings of a capture from its RAW files' EXIF: frames maps a frame index to a RawMosaic or its metadata (load_raw_metadata); each frame's settings are exposure_settings and lens_settings. The body is the frames' body_identity, which must be one (identities agree when no field both state differs, case aside; ArgumentError otherwise). The lens is lens where the caller states it — a config knows an adapted lens EXIF misnames — and the frames' one EXIF claim otherwise (lens_identity, which must then agree across the frames); a stated lens that disagrees with EXIF is used, with a warning naming both. EXIF's claim stays readable from the metadata.
VisualGeometryToolkitFeatures.lens_identity — Method
lens_identity(x) -> DeviceIdentityThe lens a RAW file CLAIMS took it (lens_make, lens_model, lens_serial). Kept, not trusted: a lens on an adapter is named by what the adapter reports (the FX3 with Sigma EF lenses claims "DT 8mm F4 SAM"), so a pipeline that knows its lens states it (extract_capture_settings's lens).
VisualGeometryToolkitFeatures.lens_settings — Method
lens_settings(x) -> LensSettingsThe lens settings a RAW frame was taken with, as Core's LensSettings: the focal length the lens reports (focal_mm, the NOMINAL value, not a calibration) and the focus distance where the file states a finite, positive one (nothing otherwise — the FX3's adapted lenses report infinity). A file without a focal length is an ArgumentError.
VisualGeometryToolkitFeatures.load_raw — Method
load_raw([T], path; mode::Symbol, gain::Real = 1, sensor_model = SensorModel(), white_balance = nothing,
decoder::RawDecoder = LibRawDecoder(), cache = nothing) -> LinearRaw
load_raw([T], mosaic::RawMosaic; mode::Symbol, gain::Real = 1, sensor_model = SensorModel(), white_balance = nothing,
decoder::RawDecoder = LibRawDecoder()) -> LinearRawA camera RAW file — or a mosaic already in memory, decoded or corrected (vignetting applied per photosite, say) — as linear values, in the sensor's orientation (the EXIF flip is never applied, in any mode). LinearRaw, RawMosaic and the frame maps are VisualGeometryToolkitCore's (exported by it). mode:
:green_cellsor:bin2— exact:load_raw_mosaic(for a file), then Core'sLinearRaw(T, mosaic; mode, gain, sensor_model), on the half-resolution cell grid thatraw_frame_mapmaps onto the raw.:green_cellsis THE image for geometry and radiometry. Rounded to 16 bits (round(UInt16, clamp(x, 0, 1) * 65535)),:bin2equals the linear PNG of RadiometricCamera's convert_raw.py (--mode bin2), value for value; unlike that PNG,saturatedflags the cells that contain a photosite at the white level.:demosaic— FOR DISPLAY FIGURES ONLY. LibRaw's demosaic (AHD) of the visible area at full resolution: it INTERPOLATES the two colours each photosite did not record, so its values are not measured and its pixels carry no exact geometry. Evaluation code must not read it; it has no frame map (raw_frame_mapandgreen_cell_sitesrefuse it). LibRaw's output is linear (gamma 1, no auto-brightening, no maximum adjustment, 16 bits, the camera's raw colour, white balance 1); thewhite_balance(an R G B multiplier triple, e.g. frommetadata.daylight_wb) and then the gain are applied here, in Julia.saturatedflags pixels LibRaw clipped. Of a mosaic in memory it is the same demosaic, LibRaw's, on the mosaic's photosites (copied to the host), its colour phase and its black and white levels (whole codes, as LibRaw holds them; anArgumentErrorotherwise): a corrected mosaic is displayed through the one demosaic, with its own metadata.
T, a Base.IEEEFloat (positional and leading, the stack's signature rule): the element type the values are stored as, in every mode, each value computed in Float64 and rounded once. The exact modes forward it to Core's LinearRaw(T, mosaic; …) (an uncertain image's variance plane is Float32 beside Float16 values and includes the store's spacing²/12); the display demosaic refuses a gain times white balance that could exceed floatmax(T), as Core does. Without it the type is Core's default (LinearRaw(mosaic; …)'s: Float64, bit for bit the values before Core 0.10.5). Reduce a Float16 image's values in Float64 (sum(Float64, A)), as every reduction in this package does.
white_balance is accepted with :demosaic only: the exact modes are camera raw colour, white balance 1.
sensor_model (the exact modes only): the frame's resolved VisualGeometryToolkitCore SensorModel (from sensor_model), handed to Core's LinearRaw, the stack's one black subtraction: the model's black (the file's, per channel, or a dark frame's per photosite), its defects joined to saturated, and under a Poisson–Gaussian or relative noise each value's variance (an uncertain image). The absent model, SensorModel() (the default), is the file's black, bit for bit. The display demosaic is LibRaw's own and takes none.
cache (a file only; a FileCache, using JLD2 provides it): each RAW is decoded once per content and settings — the entry is keyed by the file's SHA-256, mode, gain, white_balance, the element type T (the default resolved to Core's first, so a change of Core's default never serves an entry of the old type; Float64 keeps the plain key, the entries before 0.9.5 being Float64), the decoder's type and its library's version (LibRaw's, libraw_version) (cache_key), so an edited file, another setting or an upgraded LibRaw decodes again and a hit returns the LinearRaw as decoded, value for value. nothing (the default) decodes every call. A sensor model other than the absent one is part of the key by its content (Core's content_digest), so an entry is never served for another model; the absent model keeps the plain key.
VisualGeometryToolkitFeatures.load_raw_metadata — Method
load_raw_metadata(path; decoder::RawDecoder = LibRawDecoder()) -> NamedTupleThe EXIF fields of a camera RAW file, as RawMosaic's metadata holds them, without keeping the photosites: LibRawDecoder reads the headers only (no unpack). The typed accessors (exposure_settings, …) take it as they take a mosaic.
VisualGeometryToolkitFeatures.load_raw_mosaic — Method
load_raw_mosaic(path; decoder::RawDecoder = LibRawDecoder()) -> RawMosaicDecode a camera RAW file (any format LibRaw reads: ARW, CR2, NEF, DNG, …) to its mosaic, VisualGeometryToolkitCore's RawMosaic. Only flat Bayer sensors (one colour per photosite) are supported. The decoder's package must be loaded: using LibRaw_jll for LibRawDecoder.
The detection image of a RAW
variance_stabilize maps the green cells (the :green_cells image, or :bin2's green plane) to the generalized Anscombe transform for Poisson–Gaussian noise: the noise has σ ≈ 1 at every level, so ASIFT on it (asift_detect of a StabilizedImage) thresholds its DoG at noise_sigmas noise σ, in a dim interior as in a bright window, and masks the saturated pixels. A descriptor still reads a display rendering of the same cells: both are pixel-wise, so the positions coincide.
VisualGeometryToolkitFeatures.StabilizedImage — Type
StabilizedImage(image, saturated, white)A detection image whose noise has σ = 1 at every pixel: image (a Matrix{Float64}, in noise σ), the saturated pixels (a BitMatrix, masked: asift_detect returns no detection over them) and white, the image's value at the white level (the scale that maps it into [0, 1]). An image that is a Matrix{Float64} and a saturated that is a BitMatrix are held, not copied. variance_stabilize makes one of a RAW's green cells.
VisualGeometryToolkitFeatures.asift_detect — Method
asift_detect(s::StabilizedImage; noise_sigmas = 3.0, kwargs...)ASIFT on a variance-stabilized image (variance_stabilize), its DoG contrast threshold noise_sigmas noise σ: on unit iid noise, the DoG's noise σ in its noisiest searched slice (the finest octave's finest interior slice, the extremum search's first) is derived from the pyramid's own impulse response, and an extremum must exceed noise_sigmas times it. Every coarser slice, and every tilted view, has less noise, so is held to more than noise_sigmas σ of its own. The default 3 is Lowe's three-sigma reading of a contrast: on pure noise, extrema above it are rare.
The saturated pixels are the exclusion_mask (asift_detect's rule: a detection whose centre lies within three of its scales, the DoG's support, of a masked pixel is dropped): a clipped value is not a measured one. A caller's exclusion_mask joins them. The image is scaled by 1 / s.white into [0, 1] for the detector (the warps' fill is mid-grey), the threshold with it. kwargs are asift_detect's but intensity_thr (the threshold is noise_sigmas).
VisualGeometryToolkitFeatures.variance_stabilize — Method
variance_stabilize(L::LinearRaw, sensor_model::SensorModel) -> StabilizedImageThe generalized Anscombe transform of the green cells of L (:green_cells, or :bin2's G plane, the same values) for Poisson–Gaussian noise: after it the noise has σ ≈ 1 at every level, from a few electrons to the white level.
sensor_model is the frame's resolved VisualGeometryToolkitCore SensorModel (sensor_model, a lookup in the body's characterisation), the one L was linearised with (LinearRaw(mosaic; mode, sensor_model), or load_raw's sensor_model): under its Poisson–Gaussian noise (a gain) L carries each green value's variance v(x), Core's one law. A model without a gain is an ArgumentError naming the missing gain; there is no fallback. The transform reads the variance plane and the green plane's Poisson slope α (Core's poisson_slope): a value x whose variance is α·max(x, 0) + σ² (σ² its Gaussian part, a dark frame's per photosite when the model holds one) maps to (2/α)·√(max(α x + 3α²/8 + σ², 0)). white is the transform at the white level, L.gain, with the plane's mean Gaussian variance.
L.saturated is carried over: asift_detect masks those pixels rather than reading the clipped values. The transform is pixel-wise, so the detections' positions are the cell grid's, as on any rendering of L (a display rendering for a descriptor among them).
Image input, the linear PNG reader and the display encoding
A linear image is shown through one encoding, srgb_image: exposure, a white balance per plane, an optional tone curve, clamp, the sRGB transfer curve (linear_to_srgb, the inverse of srgb_to_linear) and 8 or 16 bits, a NaN pixel the background.
VisualGeometryToolkitFeatures.gray_f32 — Method
gray_f32(img) -> AbstractMatriximg as float gray: a float Gray matrix or a bare Float32 matrix (a device array already in the kernels' form) is returned UNTOUCHED — no copy, no wrap, so a CuArray{Float32} stays on the device; anything else (fixed-point gray, any colorant) converts to Gray{Float32}.
VisualGeometryToolkitFeatures.gray_u8 — Method
gray_u8(img) -> Matrix{UInt8}img as H×W gray bytes, in a Matrix{UInt8} the caller owns. A Matrix{UInt8} is returned as is; a Gray{N0f8} matrix is copied byte for byte (its pixels already are bytes); everything else is quantised through Gray{N0f8} (float colorants LOSE precision here by design — use gray_f32 where sub-pixel photometry matters).
VisualGeometryToolkitFeatures.box_downsample — Method
box_downsample(img::AbstractMatrix, f::Integer) -> Matrix
box_downsample(img::AbstractArray{T, 3}, f::Integer) -> Array{float(T), 3}The H × W image (or H × W × C, per channel) averaged over f × f blocks: output pixel (i, j) is the mean of input rows (i − 1)f + 1 : if and columns (j − 1)f + 1 : jf. The output is H ÷ f × W ÷ f (× C); rows and columns past the last whole block are dropped. The element type is a real number or a colour (Gray{Float32}, RGB{N0f8}, …); the sums accumulate, and the output is stored, in its floating-point type float(T) — Float64 for a Float64 or integer image, Float32 for a Float32 or 8-bit one (the patches BlobBoards' descriptor reads, byte for byte) — each block summed column by column and scaled once by 1/f² in that type. A Float16 image (a LinearRaw(Float16, …)'s values) is stored as Float16 but summed and scaled in Float64, each output rounded once: a Float16 sum of a block loses the values' low bits. f = 1 returns a float(T) copy.
VisualGeometryToolkitFeatures.gaussian_blur — Function
gaussian_blur(img::AbstractArray{<:Real, 3}, σ[, mask]) -> Array{Float64, 3}The H × W × C image convolved per channel with the separable Gaussian of standard deviation σ pixels, truncated at ⌈3σ⌉ pixels and normalized to unit sum; at the image border the kernel is renormalized over the pixels inside it (no padding value enters). The taps are the package's one Gaussian taps builder's, at truncation 3 — not ASIFT's pyramid's 4σ with a mirrored border: the vignetting chain this smooths for was reproduced with this truncation and border rule, so they stay as they are.
With a mask (an H × W array of weights, 0 outside the support — a fisheye's image circle) it is the NORMALIZED convolution blur(img·mask) / blur(mask): pixels outside the mask do not bleed in, and a pixel where the mask is zero is 0. σ = 0 returns a Float64 copy (masked to zero outside the mask). Edge-safe smoothing of a linear image before it is sampled (the rooftop vignetting chain samples it at fixed points).
VisualGeometryToolkitFeatures.linear_to_srgb — Method
linear_to_srgb(x) -> encoded valueThe sRGB (IEC 61966-2-1) opto-electronic transfer function, the inverse of srgb_to_linear: a linear value x ∈ [0, 1] to its display encoding, 12.92x for x ≤ 0.0031308, otherwise 1.055x^(1/2.4) − 0.055, in x's floating-point type (a Float32 image encodes in Float32). The stack's one encoder: a display image is srgb_image.
VisualGeometryToolkitFeatures.load_linear_image — Method
load_linear_image(path; srgb = false, downsample = 1) -> Array{Float64, 3}A PNG file as an H × W × 3Float64 array of linear RGB values in [0, 1]: the samples AS STORED (read with gamma = 1, so a gAMA/sRGB chunk never re-encodes them; gray files are replicated to three channels, an alpha channel is dropped, 8- and 16-bit depths both normalized to [0, 1]).
srgb = true: the file is display-encoded (a phone or camera photograph);srgb_to_lineardecodes every sample.downsample = f: the decoded image averaged overf × fblocks (box_downsample), in linear space.
The array is indexed [row, column, channel]; pixel (row, column) is the image's PixelFrame{:julia} pixel (column, row).
VisualGeometryToolkitFeatures.srgb_image — Method
srgb_image([T = N0f8], img; exposure = 1, white_balance = nothing, background = 0, tone = nothing)
-> Matrix{RGB{T}} or Matrix{Gray{T}}A linear image for display, the stack's one display encoding: an H × W × 3 image to RGB{T}, an H × W (or H × W × 1) one to Gray{T}, T a fixed-point type (N0f8, N0f16). Each value is multiplied by the exposure and by its plane's white_balance factor (one per plane, nothing: none), then mapped by the tone curve (a function of a linear value, nothing: none), clamped to [0, 1], encoded by linear_to_srgb and rounded to T. A NaN value (no measured value: a panorama pixel no frame sees) is the background, a display-linear value in [0, 1], encoded alike. The exposure and the factors are positive and finite (ArgumentError otherwise).
Runs where img lives (a broadcast: a device array is encoded on the device; tone must then be kernel-lawful).
VisualGeometryToolkitFeatures.srgb_to_linear — Method
srgb_to_linear(x) -> linear valueThe sRGB (IEC 61966-2-1) electro-optical transfer function: an encoded value x ∈ [0, 1] to linear, x / 12.92 for x ≤ 0.04045, otherwise ((x + 0.055) / 1.055)^2.4.