Camera RAW API

The sensor image: a decoded Bayer RAW as RawMosaic (an AbstractMatrix{UInt16} of its photosites that knows its colour phase, its levels and where it sits on the sensor), its photosites channel-first as a method of ImageCore's channelview (a view, on the mosaic's device), and LinearRaw, its exact linear values on the 2×2 cells, with the maps of every plane and channel onto the sensor. Decoding a file is VisualGeometryToolkitFeatures' (load_raw_mosaic, LibRaw), and so is the display demosaic. A cell mode's frame map is also read without an image, raw_frame_map(LinearRaw{:green_cells}), and its provenance record, raw_provenance(LinearRaw{:green_cells}), for a frame with no RAW behind it. See the Raw images guide.

The mosaic and its photosite views

VisualGeometryToolkitCore.BayerCellFrame — Type
BayerCellFrame{C} <: Frame{2}

The COLMAP pixel frame of one colour plane C of the half-resolution grid of 2×2 Bayer cells: pixel (row, column) of the plane is centred at (column − ½, row − ½) of its frame. The planes are indexed alike but sampled at different points of the cell, so each is a frame of its own:

  • :R, :B — a cell's red and blue photosite (a plane of a :bin2LinearRaw, a channel of a mosaic's channelview);
  • :G1, :G2 — a cell's two green photosites, in the cell's column-major order (the two green channels of channelview(m));
  • :G — the mean of the two greens, which sits at the cell centre in every phase: the cell grid itself (the :green_cells image), whose map is raw_frame_map.

bayer_plane_map maps each onto the visible raw's PixelFrame{:colmap}, and a composition that joins one plane's frame to another's is refused (MethodError), as at every frame junction of Core.

source
VisualGeometryToolkitCore.LinearRawView — Type
LinearRawView{T, A, D, L} <: AbstractMatrix{T}

The lazy linear view of a mosaic's photosites on the raw grid, linearview(m): element (i, j) is the linear value of the held array's code there, with its channel's black level and span, computed when read. Under a sensor model whose noise has a variance law (law: a PoissonGaussian or a RelativeNoise) the element is an Uncertain{V}, the value with its scalar variance computed on the same read (owner ruling D14: the default uncertain image is this lazy one; copy materialises it as a StructArray); otherwise T is V. V is the value type linearview(V, m) was asked for (Float64 by default): each element is computed in Float64 and rounded once to V (its variance to Float32 for a Float16 view; see LinearRaw, "The element type"). It holds the photosites (parent, the mosaic's held array, a device array too), the four channels' levels and, from a sensor model's Dark, the offset map over the visible raw (dark, on the same device; nothing without one) read at the mosaic's window_origin; similar, a broadcast and copy are the held array's kind (a broadcast over a device view runs on the device), adapt moves it, and a KernelAbstractions kernel reads it as it is (its indexing is two parities, the one black rule and one linearisation).

source
VisualGeometryToolkitCore.PhotositeColors — Type
PhotositeColors <: AbstractMatrix{UInt8}

The colour index of every photosite of a mosaic, lazily, as an isbits value: the size, the colour pattern (at most 4 × 4, in an NTuple) and its phase at the mosaic's first photosite. Indexing is the one phase rule; nothing is stored per photosite, so it is the same on the host and inside a GPU kernel or a fused broadcast with a device mosaic's arrays (it holds no array, and a device array in the same broadcast decides where it runs).

source
VisualGeometryToolkitCore.RawMosaic — Type
RawMosaic{A <: AbstractMatrix{UInt16}, M <: NamedTuple} <: AbstractMatrix{UInt16}

A RAW file decoded and nothing more: the visible area of the sensor, as the decoder reports it (VisualGeometryToolkitFeatures' load_raw_mosaic, LibRaw). It IS its photosite matrix: size, indexing and parent are the held array's, so an ordinary array or image function reads it unchanged, and what such a function makes (similar, copy, a broadcast, a reduction) is a plain array of the held array's kind — the colour pattern and the metadata stay with the mosaic. copy, ==, isequal and isapprox are the held array's own (no element read through the wrapper, so a device mosaic compares on the device), and equality is the photosite matrix's: two mosaics with the same photosites are == whatever their colour pattern, window origin or metadata (compare photosite_colors, window_origin and metadata for those), and a mosaic is == to its held array. It is read-only (no setindex!): write to parent(m). The held array may be a device array (adapt(CuArray, m), or adapt(backend, m) with a KernelAbstractions backend, moves the photosites and keeps the rest); a kernel receives parent(m), never the wrapper (nor, on a GPU, the nested view channelview(m): read its photosites by its index rule).

A unit-range index m[rows, columns] (a copy) or view(m, rows, columns) (no copy) is a window: a mosaic of its own whose colours are the parent's at those photosites (the colour phase shifts with an odd start, as a circshift of the pattern would) and whose raw_frame_map is its offset into the visible raw. Any other index is a plain array.

  • raw::A — the photosites, raw[row, column], in the sensor's orientation (no EXIF flip): pixel (row, column) is the raw's COLMAP pixel centred at (column − ½, row − ½) — the visible raw's PixelFrame{:colmap}, the frame every calibration of the raw is in (a window's pixels are offset by window_origin).
  • pattern::Matrix{UInt8} — the smallest repeating colour pattern, square, at most 4 × 4, from the full raw frame's origin (LibRaw's COLOR read from there), in LibRaw's colour indices: 0 R, 1 G, 2 B, 3 the second G of an RGBG pattern. Each photosite's index is photosite_colors(mosaic), the pattern tiled from that origin.
  • black::NTuple{4, Float64} — the black level per colour index (the common and per-channel parts summed, LibRaw's adjust_bl).
  • white::Float64 — the white level (LibRaw's maximum): the code at which a photosite is saturated.
  • visible_origin::NTuple{2, Int} — (top_margin, left_margin), the visible area's offset into the full raw frame, which is raw_size ((height, width)).
  • metadata::NamedTuple — what the decoder read: the EXIF fields (VisualGeometryToolkitFeatures types them: exposure_settings, lens_settings, body_identity, lens_identity, capture_time), the daylight white balance, the camera's declared linear-response limit, the decoder and its library's version. Core reads none of it except in raw_provenance.
  • window_origin::NTuple{2, Int} — (rows, columns), the offset of raw[1, 1] from the visible area's first photosite: (0, 0) as decoded (the keyword of the constructor), a window's start less one otherwise.
source
VisualGeometryToolkitCore.SaturationView — Type
SaturationView{A} <: AbstractMatrix{Bool}

The lazy saturation flags of a mosaic's photosites on the raw grid, saturationview(m): element (i, j) is raw[i, j] ≥ white, read when indexed. The match of LinearRawView: the same grid, the same held array, and the same device behaviour (similar, a broadcast and copy are the held array's kind; adapt moves it; a kernel reads it as it is).

source
ImageCore.channelview — Method
channelview(m::RawMosaic) -> AbstractArray{UInt16, 3}

ImageCore's channelview, for a Bayer mosaic: the photosites channel-first, 4 × H ÷ 2 × W ÷ 2, without a copy. The Bayer interpretation of "channels": each 2×2 cell is a pixel and its four photosites — spatially interleaved, each at its own position on the sensor — are its channels, so v[k, i, j] is photosite k of cell (i, j), the cell's photosites in column-major order (k = di + 2dj + 1 for photosite (2i − 1 + di, 2j − 1 + dj)): the mosaic's own phase order, whose colours are bayer_channels(m) (R, G1, G2, B for an RGGB cell; another phase permutes them, since reordering channels is not a view). It is built as ImageCore builds a view: a reshape of the photosite matrix (H × W → 2 × H/2 × 2 × W/2), its dimensions permuted (→ 2 × 2 × H/2 × W/2, a PermutedDimsArray) and reshaped (→ 4 × H/2 × W/2), views of parent(m): writing through it writes the mosaic's held array, a device mosaic's view is a view of its device array (KernelAbstractions' get_backend is the device's, adapt(Array, v) moves it whole), and a broadcast or a Cartesian copy over a whole mosaic's view runs on the device. An odd last row or column (no 2×2 cell) is left out, as in LinearRaw.

In a GPU kernel read the photosites by the view's index rule instead, parent(m)[2i − 1 + di, 2j − 1 + dj] (or the 4-D parent(v)[di + 1, dj + 1, i, j]): Base's ReshapedArray indexing compiles in a CUDA kernel only over a whole contiguous array, at about a third of the held matrix's bandwidth (509 against 1391 GB/s on a B200, 2848 × 4256), and not at all over a window or an odd-sized mosaic (an InvalidIRError); benchmark/raw_channelview.jl measures it, and LinearRaw's cell kernel reads the held matrix.

A plain AbstractMatrix{UInt16} keeps ImageCore's own method (a gray image's channel view is the array itself): this method is the mosaic's, whose phase says which photosite is which. Channel k's pixel frame is BayerCellFrame{C} (C its colour), and bayer_plane_map(m, k) maps it onto the visible raw, exactly, for every Bayer phase and either parity of a window's origin. Its inverse is photositeview. The linear values channel-first are channelview(linearview(m)).

source
VisualGeometryToolkitCore.bayer_channels — Method
bayer_channels(m::RawMosaic) -> NTuple{4, Symbol}

The colour of each channel of channelview(m): :R, :G1, :G2, :B, in the channels' order, the cell's photosites column-major ((0, 0), (1, 0), (0, 1), (1, 1) from the cell's top-left). (:R, :G1, :G2, :B) for an RGGB cell, (:G1, :B, :R, :G2) for GRBG; the greens are numbered in that order. The cell's phase is the mosaic's own at its first photosite, so a window with an odd start reads another phase (from the same pattern, the one phase rule of photosite_colors). A pattern that is not a 2×2 Bayer cell (one R, two G, one B) is an ArgumentError.

source
VisualGeometryToolkitCore.bayer_plane_map — Method
bayer_plane_map(m::RawMosaic, k::Integer) -> SimilarityMap{2, …, BayerCellFrame{C}, PixelFrame{:colmap}}

The map of channel k of channelview(m) onto the visible raw's PixelFrame{:colmap}: pixel (i, j) of the channel, at (j − ½, i − ½) of its frame BayerCellFrame{C} (C = bayer_channels(m)[k]), goes onto its photosite's centre, (2j − 1 + dj − ½, 2i − 1 + di − ½) plus a window's offset, exactly, for every Bayer phase and either parity of the visible or window origin. It is the cell grid's map x ↦ 2x after the photosite's translation from the cell centre ((±¼, ±¼) cell pixels): the rule the planes of a LinearRaw share, so the R channel's map IS the R plane's of the mosaic's :bin2 image. An isbits value, usable inside a kernel. A channel past the fourth is an ArgumentError.

source
VisualGeometryToolkitCore.cfa_string — Method
cfa_string(mosaic::RawMosaic) -> String

The colour pattern as letters, row by row ("RGGB" for [0 1; 3 2]): LibRaw's indices read through "RGBG", as convert_raw.py records it.

source
VisualGeometryToolkitCore.linear_quantum — Method
linear_quantum(v::LinearRawView, i::Integer, j::Integer) -> Float64

The quantum of photosite (i, j) of a linear view: the resolution of its stored value in linear units, for a caller that derives a resolution from it (an LM's residual_resolution). It is one RAW code, 1 / (white − black[c]), c its colour, the step between two adjacent codes of linearview's value there (the channel's span; a Dark replaces the black per photosite but not the span), or the spacing of the view's element type at the value, eps(value_of(v[i, j])), where that is coarser: a Float16 view near the white level steps by 2⁻¹¹, span/2048 codes (≈ 7.75 on a 14-bit span). For the Float64 default the spacing is always finer (eps(x) < 1/65535 for every value), and the quantum is the code's, as before.

source
VisualGeometryToolkitCore.linearview — Method
linearview([T = Float64], m::RawMosaic; sensor_model::SensorModel = SensorModel()) -> LinearRawView

The mosaic's linear values on the raw grid, lazily, as ImageCore's normedview reads codes as numbers without a copy: element (i, j) is (raw − black[c]) / (white − black[c]) of photosite (i, j), c its colour — the one linearisation, the one LinearRaw's cell kernel applies. Not clipped, no gain. It composes with ImageCore's channelview: channelview(linearview(m)) is the linear values channel-first (4 × H/2 × W/2, the channels of channelview(m)), so the green-cell value of cell (i, j) is the mean of its two green channels, bit for bit (LinearRaw(m; mode = :green_cells).image[i, j, 1], at gain 1). An odd last row or column is in the view (its photosites' colours by the same rule) and left out of the channel view.

Not normedview(N0f16, …) itself, whose code / 65535 ignores the black levels, nor a MappedArrays.mappedarray of the codes, whose function sees the value alone: the black level and span depend on the photosite's position (its colour in the phase), so the view reads its index — one parity rule, the cell sites' (k = di + 2dj + 1).

sensor_model, the frame's SensorModel, sets the levels as it does LinearRaw's, by the same resolution: its black slot's per-colour-index black (the span white − black[c]), and a Dark's offset in place of the black per photosite. Its defects are not values: they join LinearRaw's mask, not this view. Its noise is: under a PoissonGaussian or a RelativeNoise each element is an Uncertain{T}, the value and its scalar variance (Statistics.var(x), computed on the read by the law LinearRaw's kernel writes per plane, one site at gain 1: owner ruling D14, the default uncertain image of the photosites; the binned cell image, LinearRaw's, is materialised once by its kernel and never re-derived by a consumer), and copy of the view is the materialised form, a StructArray of a value plane and a variance plane (on the view's device). The layout is the raw grid, H × W; channelview of it is the 4 × H/2 × W/2 cells, uncertain alike. With the absent model, the default, the view is the mosaic's own levels, plain values.

T (a Base.IEEEFloat) is the elements' value type, as LinearRaw's (the same one default): each element is computed in Float64 and rounded once to T, so copy(linearview(Float16, m)) is Float16.(copy(linearview(m))) bit for bit, and an uncertain element's variance is Float32 beside a Float16 value. A lazy view stores nothing; the type is what a copy, a broadcast or a kernel reading it holds.

source
VisualGeometryToolkitCore.photosite_colors — Method
photosite_colors(mosaic::RawMosaic) -> PhotositeColors <: AbstractMatrix{UInt8}

Each photosite's colour index (LibRaw's: 0 R, 1 G, 2 B, 3 the second G), the size of mosaic: the pattern tiled from the full raw frame's origin, so photosite (row, column) reads pattern[mod(top + row − 1, n) + 1, mod(left + column − 1, n) + 1] with (top, left) = visible_origin .+ window_origin. Lazy and isbits (PhotositeColors): no matrix is built, and it broadcasts with a device mosaic's arrays on the device, fused, so a photosite mask is one broadcast with no host round-trip:

col = photosite_colors(m)
valid = ((col .== 0x01) .| (col .== 0x03)) .& (parent(m) .!= 0) .& .!saturationview(m)

(collect gives a Matrix{UInt8}.)

source
VisualGeometryToolkitCore.photositeview — Method
photositeview(v::AbstractArray{T, 3}) -> AbstractMatrix{T}

The inverse of a mosaic's channelview, in the role ImageCore's colorview plays for an image's (a name of its own: colorview makes colorants of channels, and this makes the photosite matrix of cells): a channel-first array of 2×2 cells (4 × h × w, each column a cell's photosites in column-major order) as the 2h × 2w photosite matrix, without a copy. Of a mosaic's channelview it is the mosaic's (even-sized) held array again; of any other array (per-photosite values computed channel-first, say) it is a view in the mosaic's layout, photositeview(v)[2i − 1 + di, 2j − 1 + dj] == v[di + 2dj + 1, i, j]. A first dimension other than 4 is an ArgumentError.

source
VisualGeometryToolkitCore.raw_frame_map — Method
raw_frame_map(m::RawMosaic) -> SimilarityMap{2, …, PixelFrame{:colmap}, PixelFrame{:colmap}}

The map of the mosaic's pixels onto the visible raw's PixelFrame{:colmap}: the identity as decoded, a window's offset (x ↦ x + (columns, rows) of its window_origin) for a window.

source
VisualGeometryToolkitCore.saturationview — Method
saturationview(m::RawMosaic) -> SaturationView

The mosaic's saturation flags on the raw grid, lazily: true at a photosite whose code is at or above the white level, raw ≥ white with no margin — THE saturation rule, the one LinearRaw's flags apply per cell (a cell is flagged when any photosite its planes read is; under a sensor model holding a DefectList LinearRaw also masks those photosites). The Bool match of linearview(m): the same grid, and channelview of it is the flags channel-first. A mask of the usable photosites of one colour is a plain broadcast with photosite_colors:

col = photosite_colors(m)
usable_green = ((col .== 0x01) .| (col .== 0x03)) .& .!saturationview(m)
source

Linear values on the 2×2 cells

VisualGeometryToolkitCore.LinearRaw — Type
LinearRaw

A RAW image as linear values: (raw − black[c]) / (white − black[c]) × gain (each photosite's linearview value), camera raw colour, white balance 1, 1.0 = the white level (before the gain). Not clipped: a value above 1 (a gain above 1) or below 0 (noise under the black level) is kept.

  • image — image[row, column, plane], on the mosaic's device (an Array on the host); a plane's frame is asked of the image with its index (bayer_plane_map(L, c)). Under a sensor model whose noise defines a variance (PoissonGaussian, RelativeNoise) it is a StructArray of Uncertain{T} values (owner ruling D11): its value component the values of element type T (below), its attributes component each value's variance as a 1×1 CovMat (cov(L.image[i, j, p])[1], of Float32 for Float16 values, else of T), both planes from the one kernel. This cell image is the MATERIALISED uncertain image (owner ruling D14, (a)): its binned variances are computed once, in the one kernel, from the one variance formula, and a consumer never re-derives them; the photosites' uncertain image, linearview(m; sensor_model), is the lazy default. Otherwise (no noise, a ReadNoise alone) it is a plain AbstractArray{T, 3}. A consumer dispatches on the image's element type.

  • saturated::AbstractMatrix{Bool} (a BitMatrix on the host) — the MASKED pixels: those whose value includes a photosite at the white level (raw ≥ white) or, under a sensor model holding a DefectList, a defective photosite: in :bin2 a cell is masked when ANY of its four photosites is, so a cell whose two greens straddle the white level is flagged rather than averaged into a plausible value; in :green_cells when either green is. Masked, never interpolated.

  • geometry — where the samples are, (; plane_maps, green_offsets, plane_sites):

    • plane_maps, per plane, its map onto the visible raw (bayer_plane_map, which reads it);
    • green_offsets, the cell's two green photosites, (Δx, Δy) from the cell centre in RAW pixels, in the cell's column-major order (G1, G2 of bayer_channels; green_cell_sites);
    • plane_sites, per plane, the photosites its value is the mean of, each (; color, dn): its CFA colour index (LibRaw's 0 R, 1 G, 2 B, 3 the second G; the index of a per-colour component's entry, σ[color + 1]) and its white − black[c], the codes that span 0 … 1 (before the gain) — what a value's photon and read noise scale by. The green plane's are in the order of green_offsets. black[c] is the channel's constant (the sensor model's black slot's) also when a Dark replaces it per photosite: the span is a scale, the same for every photosite of the channel, so a value is still codes / dn above its own black.

    nothing for :demosaic, whose pixels are interpolated, not samples.

  • gain (positive and finite, ArgumentError otherwise) and the mosaic's metadata (the decoder and its library's version among it).

The element typeT of the values is chosen at construction, LinearRaw(T, mosaic; mode): Float64 by default (one constant, _LINEAR_RAW_ELTYPE), Float32 or Float16. Each value is computed in Float64 (the black subtraction, the cell's mean, the gain) and rounded ONCE, to nearest, when it is stored, and its variance is computed from the unrounded value, plus the variance the store adds, spacing²/12 at the value (0 for Float64: the stored value's variance, see the bright end below); so LinearRaw(Float16, m; mode).image == Float16.(LinearRaw(m; mode).image), bit for bit, and each value is within eps(Float16)·|x|/2 of the Float64 one (half the subnormal step, 2⁻²⁵, below 2⁻¹⁴). The mosaic stays its UInt16 codes. Why Float16 holds a RAW's values (the Sony FX3: white 16383, black 512, so the span is d = 15871 codes, and K = 11.33 e⁻/code):

  • The bright end. A value is its codes above black over d. Float16 keeps 11 significant bits, so on [½, 1) — 7936 to 15871 codes above black — its step is 2⁻¹¹·d = d/2048 ≈ 7.75 codes, about 8. The shot noise there is σ = √(μ/K) codes: √(16000/11.33) ≈ 37.6 at 16k codes and √(7936/11.33) ≈ 26.5 at the binade's bottom. The rounding adds its variance, step²/12 ≈ 5.0 codes² per value, which an uncertain image's variance plane includes (a consumer that pools the stored values' spread, a photon-transfer fit over Float16 values, sees it: on the FX3 it lowers K from 11.30 to 11.18 e⁻/code, about 22 times K's counts standard error, the added variance per point measured over step²/12 0.98 to 1.03, and 11.297 with it subtracted). Against the shot variance μ/K that is largest at the bottom of the binade, μ = d/2: (d/2048)²·K/(12·d/2) = d·K/(6·2048²) ≈ 0.71% of the variance (0.36% of σ). Each binade below halves it: the step halves, so its square quarters, while μ only halves.
  • The darks. Near the black level Float16 is relative: 2⁻¹¹ ≈ 4.9e-4 of the value, log₁₀ 2048 ≈ 3.3 significant digits, and below 2⁻¹⁴ (d·2⁻¹⁴ ≈ 0.97 codes) a fixed step of 2⁻²⁴·d ≈ 9.5e-4 codes. A 16-bit integer PNG of the codes held whole codes, a step of 1.
  • The variance plane is not Float16 (Float32 beside Float16 values). In the values' units² a dark green cell's variance, its read noise alone, is σ_r²/(2d²) (two greens, gain 1): ≈ 7.9e-9 for σ_r = 2 codes. That is below half Float16's smallest subnormal (2⁻²⁴ ≈ 6.0e-8), so it would be stored as 0, and a whitened row would divide by it. At the white level the shot variance 1/(2Kd) ≈ 2.8e-6 is subnormal (below 2⁻¹⁴ ≈ 6.1e-5), held to about 2% (its spacing 6.0e-8). Float32's smallest normal is 1.2e-38.

A Float16 image takes 2 bytes per value (6 with its variance) against 8 (16). A value must be finite in T: a gain at which a code could exceed floatmax(T) (65504 for Float16: a gain past ≈ 16000 on a 14-bit span) is refused (ArgumentError). Reduce it in Float64: sum(Float64, A), mean(Float64, A). A Float16 accumulation overflows past 65504 (a million green cells near the white level sum to ≈ 8e5) and drops low bits long before. Core's estimators (master_dark, photon_transfer, defect_onsets) read the mosaics' UInt16 codes and accumulate in Float64, so the element type never reaches them.

The mode is the type's first parameter, LinearRaw{:green_cells}, LinearRaw{:bin2} or LinearRaw{:demosaic}: the geometric accessors are methods of the cell images, and the display image's are its refusals. raw_provenance gives the mode, the gain, the metadata and the frame map as plain data for a result to record.

source
VisualGeometryToolkitCore.LinearRaw — Method
LinearRaw([T = Float64], mosaic::RawMosaic; mode::Symbol, gain::Real = 1, sensor_model::SensorModel = SensorModel()) -> LinearRaw

The mosaic's linear values, (raw − black[c]) / (white − black[c]) × gain, on the half-resolution grid of 2×2 Bayer cells (H ÷ 2 × W ÷ 2; an odd last row or column is dropped), arranged by mode (required: there is no neutral choice between them):

  • :green_cells — THE image for geometry and radiometry, one plane: each cell's two green photosites' mean. The greens of a cell are always diagonal, so in every Bayer phase their mean sits at the cell centre (offset (0, 0)), and the cell grid maps onto the raw exactly by raw_frame_map (x ↦ 2x in COLMAP coordinates). The footprint of a value is its two photosites, at (±½, ∓½) raw pixels from the cell centre for an RGGB or BGGR cell, (±½, ±½) for GRBG or GBRG (green_cell_sites).
  • :bin2 — the same cells with their red and blue, × 3 planes: R, the mean of the two greens (the :green_cells image, bit for bit), B. R and B sit at their photosites, (±¼, ±¼) cell pixels from the centre: each plane's map onto the raw is bayer_plane_map.

Every value is from the mosaic's linearview, read channel-first (channelview(linearview(mosaic)), the mosaic's channelview): the green-cell value of cell (i, j) is the mean of the view's two green channels there, times the gain, bit for bit (the Float64 view's, at the Float64 default; for another element type T the value is that Float64 mean rounded once to T, not the mean of two rounded values). T, a Base.IEEEFloat (Float64, Float32, Float16), is the values' storage: see LinearRaw, "The element type", for the default, the rounding and why Float16 suffices. :demosaic (LibRaw's interpolation, for display figures) is VisualGeometryToolkitFeatures': load_raw(path or mosaic; mode = :demosaic).

The values are those convert_raw.py writes (--mode bin2), in the same Float64 arithmetic and order, before its clip to [0, 1] and 16-bit rounding.

THE black subtraction is here, once, for every caller: sensor_model, the frame's SensorModel (sensor_model resolves it from a characterisation), each slot by its component:

  • black: FileBlack (the absent model's) the mosaic's own per-channel mosaic.black — with the absent model SensorModel(), the default, bit for bit the values without the keyword; ChannelBlack its values (the span white − black[c]); a Dark its per-photosite offset in place of the channel constant in the subtraction ((raw − offset) / (white − black[c]), black its channel black: the per-channel scalar is the constant-map case of the same kernel path);
  • defects: a DefectList's photosites join the saturation flags (saturated is then "masked: saturated or defective"); NoDefects adds none;
  • noise: PoissonGaussian or RelativeNoise make image an uncertain image, each value with its variance from the same kernel (D11): g·max(x, 0)/(m²K)·Σ 1/d + g²/m²·Σ s/d² for a value x of m photosites (spans d, gain g, s the photosite's σ_r[c]², or under a Dark its dark variance times 1 + 1/frames: its read noise and dark current, and the error of the offset subtracted), (σ_rel·x)² under a RelativeNoise; NoNoise and ReadNoise leave it plain (a read-noise variance alone underestimates a well-exposed pixel's, ruling A).

The maps cover the visible raw and are read at the mosaic's window_origin (DimensionMismatch when they do not reach the mosaic's photosites); they are moved to the mosaic's device.

Saturation is raw ≥ white with no margin: on the Sony ARW sets this was written for, codes pile up exactly at the white level (16383) while the histogram runs smoothly through the camera's declared linear limit (15360, metadata.linear_max), so any margin would be a tuned constant. A caller who wants that stricter limit thresholds mosaic.raw itself.

source
VisualGeometryToolkitCore.bayer_plane_map — Method
bayer_plane_map(L::LinearRaw, c::Integer) -> SimilarityMap{2, …, BayerCellFrame{C}, PixelFrame{:colmap}}

The Bayer transform of plane c of the image: the map of the plane's own frame (BayerCellFrame{C}, C its colour — :R, :G, :B for :bin2, :G for :green_cells) onto the visible raw's PixelFrame{:colmap}. It is the plane's translation onto the cell grid (where its samples sit from the cell centre: (±¼, ±¼) cell pixels for R and B, none for G) composed with raw_frame_map(L), so pixel (row, column) of the plane, at (column − ½, row − ½) of its frame, goes onto the centre of the photosites its value is the mean of — R's or B's one, the two greens' midpoint — exactly, for every Bayer phase and either parity of the visible origin. The green plane's map IS raw_frame_map(L), and the R and B planes' maps are the R and B channels' of the mosaic's channelview (bayer_plane_map(m, k)): one rule.

A warp reads plane c of an image through inv(bayer_plane_map(L, c)) ∘ math, math landing in the raw's pixels, like any other map (image_warp; one warp per plane: image_warp!(dst, warps, L.image) with a tuple of them). A composition that joins two planes' frames is refused (MethodError). A plane past the image's, or a :demosaic image, is an ArgumentError.

source
VisualGeometryToolkitCore.green_cell_sites — Method
green_cell_sites(L::LinearRaw, row::Integer, column::Integer) -> NTuple{2, SVector{2, Float64}}

The footprint of the green-cell value of pixel (row, column): the centres of its two green photosites in the visible raw's COLMAP frame, G1 then G2 (the cell's column-major order). They are diagonal, at (±½, ∓½) raw pixels from the cell centre raw_frame_map(L)((column − ½, row − ½)) in an RGGB or BGGR cell and (±½, ±½) in a GRBG or GBRG one, and the value is the mean of the two — what a renderer models to predict it exactly. Each is the green channel's bayer_plane_map of the mosaic at that pixel. A :demosaic image is refused (ArgumentError).

source
VisualGeometryToolkitCore.green_plane — Method
green_plane(L::LinearRaw) -> AbstractMatrix (of the image's element type)

The image's green cells, a view: each cell's two greens' mean — L.image's plane whose frame IS the cell grid (raw_frame_map(L)): a :green_cells image's one plane, a :bin2 image's second (G). The plane every green-cell consumer reads (a detector, VisualGeometryToolkitFeatures' variance_stabilize); its photosites' colours and spans are L.geometry.plane_sites of that plane. An uncertain image's green plane is uncertain too (a StructArray view: values and variances). A :demosaic image is refused (ArgumentError).

source
VisualGeometryToolkitCore.poisson_slope — Method
poisson_slope(L::LinearRaw, model::SensorModel) -> NTuple{P, Float64}

Per plane of L (L.image[:, :, p]), the Poisson slope α of a value's variance under model: the α of α·x + σ², gain/(m²K)·Σ 1/d over the plane's m photosites (L.geometry.plane_sites[p]), 0 without a Poisson term (a model whose noise is not PoissonGaussian). The one expression the kernel writes the variance plane with; the generalized Anscombe transform reads it beside the variance plane (VisualGeometryToolkitFeatures' variance_stabilize). A :demosaic image is refused (ArgumentError).

poisson_slope(L::LinearRaw, model::SensorModel, plane) -> Float64

The slope of one plane: plane its index p, or its colour, :R, :G or :B (:G the green plane, green_plane(L)'s; a :green_cells image has :G alone). A colour the image does not hold is an ArgumentError.

source
VisualGeometryToolkitCore.raw_coordinates — Method
raw_coordinates(img, x) -> SVector{2, Float64}

Where the point x of an image's pixel frame (COLMAP, (column − ½, row − ½) the centre of pixel (row, column); any 2-vector or tuple) lies on the sensor: in the visible raw's PixelFrame{:colmap}, the frame of every calibration of the raw. THE accessor of an image's sensor-pixel coordinates, raw_frame_map(img)(x):

  • a RawMosaic: the identity as decoded, a window's offset added for a window;
  • a LinearRaw cell image (:green_cells, :bin2): the cell grid's map, x ↦ 2x (and a window's offset) — a detection on the green cells reports its position through it, never as raw pixels unmapped (half the scale); a position on another plane of a :bin2 image is bayer_plane_map(L, c)(x), and on a channel of a mosaic's channelview(m) it is bayer_plane_map(m, k)(x);
  • a :demosaic image is refused (ArgumentError), and a plain matrix, which states no frame, has no method.
source
VisualGeometryToolkitCore.raw_frame_map — Method
raw_frame_map(L::LinearRaw) -> SimilarityMap{2, …, BayerCellFrame{:G}, PixelFrame{:colmap}}

THE map of the image's pixel frame (the cell grid, which is its green plane's frame) onto the visible raw's PixelFrame{:colmap} (the frame of RawMosaic.raw and of every calibration of the raw): x ↦ 2x, exactly, for every Bayer phase and either parity of the visible origin (the cells start at the visible area's first photosite; an odd margin changes only the cell's colour phase); for an image made from a window of the mosaic, x ↦ 2x then the window's offset. Its inverse is inv(raw_frame_map(L)), and a calibration matrix moves with it through Core, reframed_calibration(raw_frame_map(L), K_cells), K_cells in the cell image's pixels (BayerCellFrame{:G}), giving K in the raw's pixels (and back with the inverse; K carries no frame label, so which frame it is in is the caller's statement). A :demosaic image is refused (ArgumentError): it is for display.

source
VisualGeometryToolkitCore.raw_frame_map — Method
raw_frame_map(::Type{LinearRaw{mode}}) -> SimilarityMap{2, …, BayerCellFrame{:G}, PixelFrame{:colmap}}

The frame map of a mode image made from a whole mosaic (no window), without an image: x ↦ 2x, exactly, for :green_cells and :bin2 alike (the cell grid; a map that depends on the mode alone, not on the photograph, its Bayer phase or its visible origin). THE source of the cell images' map: every LinearRaw is made with this map after its mosaic's own (raw_frame_map(mosaic), a window's offset), so raw_frame_map(LinearRaw(m; mode)) == raw_frame_map(m) ∘ raw_frame_map(LinearRaw{mode}) bit for bit, and for an unwindowed mosaic the two are equal. For a frame with no RAW behind it (a decoded image on the cell grid) whose map is needed, e.g. scale(raw_frame_map(LinearRaw{:green_cells})) == 2. LinearRaw{:demosaic} is refused (ArgumentError), as its images are.

source
VisualGeometryToolkitCore.raw_provenance — Method
raw_provenance(L::LinearRaw) -> NamedTuple

How the image was made, as plain data a result records: decoder, libraw_version (from the mosaic's metadata; nothing where it states none), mode, gain and frame_map ((; from, to, scale, translation) of raw_frame_map(L), nothing for :demosaic). Its mode is the mode's own record's (raw_provenance(LinearRaw{mode})), and both records' frame_map are written by one rule from the map, so an image of a whole mosaic records the mode's frame_map; a window's records its offset in translation.

source
VisualGeometryToolkitCore.raw_provenance — Method
raw_provenance(::Type{LinearRaw{mode}}) -> (; mode, frame_map)

The part of raw_provenance(L) that the mode alone determines, without an image: mode ("green_cells", "bin2", "demosaic") and frame_map, the record of raw_frame_map(LinearRaw{mode}) ((; from, to, scale, translation); nothing for :demosaic, which holds no geometry). An image made from a whole mosaic (no window) records the same mode and frame_map, == field for field, whatever its Bayer phase or visible origin. For a result whose frames have no RAW behind them that records the frame they are on.

source

The sensor model: the black level, the noise and the defects

Three levels: the geometric Sensor (part of a camera), a SensorCharacterisation (ONE body's components per ISO, held beside the camera like CaptureSettings) and a SensorModel (ONE frame's black, noise and defect components). Each slot's component is a type, absence included (FileBlack, NoNoise, NoDefects), and each present one states its source (SensorSource: measured, estimated, specified, file). The resolver sensor_model is a lookup: the frame's body and ISO pick an entry, and with none the absent model SensorModel() linearises bit for bit as the RAW alone. LinearRaw(mosaic; mode, sensor_model) is THE one black subtraction; under a PoissonGaussian or RelativeNoise its image carries each value's variance. The estimators build a characterisation's components from frames, offline: photon_transfer, master_dark, defect_onsets.

VisualGeometryToolkitCore.ChannelBlack — Type
ChannelBlack(values, std_error = nothing; source::SensorSource.T)

A per-channel black level: values per CFA colour index (LibRaw's 0 R, 1 G, 2 B, 3 the second G: values[c + 1], RawMosaic.black's layout), in codes, finite; std_error its standard error per channel (nonnegative), or nothing where none is stated (a specified value). source is measured, estimated or specified (the RAW's own is FileBlack). Every value is below the white level where it is used (ArgumentError there).

source
VisualGeometryToolkitCore.Dark — Type
Dark(black, offset, variance, exposure_band, frames; source::SensorSource.T)

A master dark of one body at one ISO, for one exposure-time band — the black slot's per-photosite form (D2: all dark information in ONE component):

  • black — its per-channel black (ChannelBlack or FileBlack): the span white − black[c] of every photosite of the channel, and the black a frame whose exposure the band does not serve is linearised with;
  • offset — per photosite of the visible raw (raw[row, column], the size of the decoded mosaic), the mean code of the lens-capped frames (the black, the DSNU and the dark current together): LinearRaw's kernel subtracts it in place of the channel's black;
  • variance — its temporal variance (codes²: the read noise with the rounding, and the dark current's shot noise), the Gaussian part of a PoissonGaussian model's variance at that photosite;
  • exposure_band = (shortest, longest), time quantities, the exposures it serves (stored in seconds);
  • frames, the number of frames averaged (at least 2).

Made by master_dark (SensorSource.measured). Moved to a device by Adapt.adapt, as LinearRawView is.

source
VisualGeometryToolkitCore.Darks — Type
Darks(black, darks; source::SensorSource.T)

The master darks of one body at one ISO, one Dark per exposure band — the black slot's form for darks taken at several exposures (Core 0.10.3):

  • black — the shared per-channel black (ChannelBlack or FileBlack), the black a frame whose exposure NO band serves is linearised with;
  • darks — the Darks, any order, stored in increasing band; their bands are DISJOINT (two bands sharing an exposure, an endpoint included, are an ArgumentError: which would serve it?), their maps of one size, at least one, each of source.

sensor_model resolves a frame to the ONE Dark whose band serves its exposure (Matched: the model is the one that Dark alone gives), else to black with the reason DarkOutOfBand. The selection is inside the one component: the slot's precedence of sources is unchanged. A single band is a Darks of one; a bare Dark stays a Dark. Each Dark keeps its own channel black (its span when it serves a frame: the black measured at its exposure); black is the fallback alone. A Darks is a characterisation's, never a frame's: a SensorModel refuses one (resolve it with sensor_model), and the model sensor_model returns, holding one Dark, is what moves to a device (Adapt.adapt of a Darks moves every band's maps).

source
VisualGeometryToolkitCore.DefectList — Type
DefectList(x, y, onset; size, source::SensorSource.T)

The defective photosites of a sensor: photosite (x[i], y[i]) — x the column and y the row of the visible raw, 1-based (raw[y, x]) — is a defect at every exposure of at least onset[i] (a time quantity, stored in seconds: the shortest exposure at which it is hot). size = (rows, columns) is the visible raw's (every site inside it, each listed once). Made from per-exposure masks by defect_onsets. sensor_model keeps the sites whose onset is at most the frame's exposure, and LinearRaw masks them (masked, never interpolated). Moved to a device by Adapt.adapt.

source
VisualGeometryToolkitCore.FileBlack — Type
FileBlack()

The black slot's absent default: the RAW file's own per-channel black level (RawMosaic.black, LibRaw's), today's behaviour bit for bit. Its source is SensorSource.file.

source
VisualGeometryToolkitCore.ModelReason — Type
ModelReason

Why sensor_model gave a frame the model it did (an isbits @enum, SensorModel.reason):

  • Matched: a characterisation of this body holds an entry at the frame's ISO; its components are the model's (an absent slot of the entry is the absent type);
  • NoCharacterisation: none was given (also the absent default SensorModel());
  • UnmatchedISO: characterisations of this body exist, none at this ISO (no extrapolation across ISO);
  • OtherBody: the characterisations given are of other bodies;
  • DarkOutOfBand (Core 0.10.2): the body and the ISO matched, but the frame's exposure is outside the entry's Dark's exposure band, so the model's black is the Dark's ChannelBlack (the entry's other components kept): the fallback is visible, never silent. Of Darks (0.10.3): no band serves the exposure, and the model's black is their shared channel black.

NoCharacterisation, UnmatchedISO and OtherBody give the absent model; Matched and DarkOutOfBand the entry's components. A caller that matches the values exhaustively handles DarkOutOfBand too.

source
VisualGeometryToolkitCore.PoissonGaussian — Type
PoissonGaussian(gain, σ, std_error = nothing; source::SensorSource.T)

The Poisson–Gaussian noise: gain K, electrons per code (positive, finite), and the read noise σ per CFA colour index, in codes, the rounding included; a photosite's variance about its black-subtracted mean μ (codes) is μ/K + σ[c+1]². std_error is (; gain, σ) (a number and a 4-tuple) or nothing. source is measured, estimated (photon_transfer) or specified. Under it a LinearRaw image carries each value's variance (an Uncertain image).

source
VisualGeometryToolkitCore.ReadNoise — Type
ReadNoise(σ, std_error = nothing; source::SensorSource.T)

The read noise alone: σ per CFA colour index (σ[c + 1]), in codes, the rounding included (nonnegative, finite), and its standard error per channel or nothing. Stored and recorded, but it defines no pixel variance: a read-noise variance alone badly underestimates the noise of a well-exposed pixel, so a LinearRaw image under it is plain, as under NoNoise (ruling A).

source
VisualGeometryToolkitCore.RelativeNoise — Type
RelativeNoise(σ_rel; source::SensorSource.T)

A noise proportional to the value: a value x has variance (σ_rel·x)² (σ_rel positive, finite). Its source is SensorSource.specified only, stated by a person (owner ruling on D14): no estimator or measurement makes one, so any other source is an ArgumentError naming it — the one rule, at the source (an IO reader of a stored one meets it here too). The model of a processed (tone-mapped) intensity, not of the photosites' physics; a LinearRaw image under it carries each value's variance (an Uncertain image), with no read-noise or dark term.

source
VisualGeometryToolkitCore.SensorCharacterisation — Type
SensorCharacterisation(body::DeviceIdentity; entries)

The description of ONE physical sensor, host-side: body, the key (serial if known, else make and model), held beside the camera like CaptureSettings, never in a camera or lens model; and entries, per ISO its components — an iterable of iso => (; black, noise, defects), each slot optional (an absent slot IS the absent type: FileBlack, NoNoise, NoDefects), so a partial characterisation is a mix of components. c.entries is a Dict{Float64, SensorEntry} (ISO → the three slots). An ISO listed twice, a slot not among black, noise and defects (a dark map is the black slot's Dark), or a non-positive ISO is an ArgumentError.

sensor_model resolves a frame's model from it. Written and read by VisualGeometryToolkitIO's vgio/sensor_characterisation kind.

source
VisualGeometryToolkitCore.SensorModel — Type
SensorModel(black::BlackComponent, noise::NoiseComponent, defects::DefectComponent;
            reason::ModelReason = Matched, specified_override::Bool = false)
SensorModel()

The sensor model of ONE frame: the black, noise and defects components sensor_model picked for its ISO and exposure (a Dark only when its band serves the exposure, a DefectList holding only the sites whose onset is at most the exposure), reason why (a ModelReason) and specified_override, whether a specified component displaced a measured one (the run record notes the override).

SensorModel() is the absent model, SensorModel(FileBlack(), NoNoise(), NoDefects(); reason = NoCharacterisation): LinearRaw and linearview under it are bit for bit the RAW's own linearisation. A model of isbits components is isbits; one holding a dark map or a defect list moves to a device with Adapt.adapt. content_digest is its identity.

source
VisualGeometryToolkitCore.component_sources — Method
component_sources(m::SensorModel) -> NamedTuple{(:black, :noise, :defects, :reason, :specified_override)}

Where each of a frame's sensor components came from — the provenance a run record states (ruling A): black, noise and defects each the component's SensorSource.T (a FileBlack's is file), or nothing for an absent slot (NoNoise, NoDefects); reason why the frame got this model (a ModelReason: Matched, NoCharacterisation, UnmatchedISO, OtherBody, or DarkOutOfBand when the entry's Dark did not serve the frame's exposure (or no Dark of its Darks did) and the ChannelBlack stands in); specified_override whether a specified component won over a measured one. The one public view of a model's provenance: a consumer reads it, never the components' fields.

source
VisualGeometryToolkitCore.content_digest — Method
content_digest(model::SensorModel) -> String

The identity of a frame's sensor model: the SHA-256 (lower-case hex) of the canonical JSON (canonical_json, the stack's one content-identity encoding) of a tree of its components — each component's type, values, standard errors and source, a map (a Dark's offset and variance) by the SHA-256 of its Float32 values' bytes and its size, a DefectList by its sites — its reason and specified_override. Equal models have equal digests, on any device (a device map is read to the host); a cache keyed by a model (VisualGeometryToolkitFeatures' decode cache) keys on it.

source
VisualGeometryToolkitCore.defect_mask — Method
defect_mask(model::SensorModel, mosaic::RawMosaic; backend = the mosaic's) -> AbstractMatrix{Bool} or nothing
defect_mask(defects::DefectComponent, mosaic::RawMosaic; backend = the mosaic's)

The defect slot as a mask over the VISIBLE raw (raw[y, x], the DefectList's size), on backend: true at each site of the list (sensor_model has cut a model's to the frame's exposure), the mask LinearRaw joins its saturation mask with. nothing for NoDefects: no photosite is masked. Indexed by the visible raw, not the mosaic's window: a window's photosite (i, j) is mask[i + o[1], j + o[2]], o = mosaic.window_origin. A list that does not cover the mosaic's photosites is a DimensionMismatch.

source
VisualGeometryToolkitCore.sensor_model — Method
sensor_model(characterisation, capture::CaptureSettings, t::Integer) -> SensorModel

THE resolver of frame t's SensorModel: a LOOKUP (owner ruling D1; it never estimates — photon_transfer makes an estimated entry offline). characterisation is a SensorCharacterisation, a vector or tuple of them, or nothing; the frame's body is capture.body, its ISO and exposure capture[t].exposure.

  1. The characterisations of this body (serial if both state one, else make and model; DeviceIdentity), and their entries at exactly the frame's ISO (no extrapolation across ISO).
  2. Per slot, independently, the component those entries hold; where several hold one, the specified one over the measured over the estimated (D5), and specified_override records a specified one displacing a measured one; two of the same source are an ArgumentError, as are two different bodies matching a capture that states no serial.
  3. A Dark only when its exposure band serves the frame's exposure (otherwise its channel black, and the reason DarkOutOfBand); of Darks, the one Dark whose band serves it (otherwise their shared channel black, and DarkOutOfBand); a DefectList cut to the sites whose onset is at most the exposure (none left: NoDefects).

With no characterisation (NoCharacterisation), none of this body (OtherBody) or none at this ISO (UnmatchedISO) the result is the absent model SensorModel(FileBlack(), NoNoise(), NoDefects()) with that ModelReason: today's linearisation, bit for bit. Otherwise its reason is Matched, or DarkOutOfBand when the entry's Dark does not serve the frame's exposure (or no Dark of its Darks does). A frame the capture does not hold is an ArgumentError. An explicit model a caller holds is used as it is: this resolver is where a characterisation becomes one, never a substitute for one given.

source
VisualGeometryToolkitCore.value_variance — Method
value_variance(v::LinearRawView, i::Integer, j::Integer, x::Real) -> Float64

The variance the view's noise law gives a linear value x at photosite (i, j) (gain 1, the law linearview's element there carries at its own value, the store's rounding aside): for a CENSORED photosite — saturated, its value clipped at the white level — the variance of that clipped value, value_variance(v, i, j, 1.0), never 0 under a law. 0 for a view without a variance law (no noise, or read noise alone: a plain view). Kernel-lawful.

source
VisualGeometryToolkitCore.PhotonTransferMetadata — Type
PhotonTransferMetadata

The FitResult metadata of photon_transfer(groups, darks) (its estimate a PoissonGaussian), read through the result's forwarded properties:

  • level_range::NTuple{2, Float64} — the fit's lowest and highest point (block means, codes above black); level_fraction the same as fractions of the white level above black (the greens' span);
  • intercept::Float64 — σ₀² (codes²), the variance at zero signal, fixed from the darks at each group's own exposure (the median over the fitted points);
  • curvature, curvature_t — the μ² coefficient c of v = σ₀² + μ/K + cμ² over the range and its t (c over its standard error);
  • tests, test_level — the μ² tests the lower end's search can perform (m, the candidate lower ends) and each test's two-sided level, significance / m (Bonferroni: the search's family-wise rate is significance); rollover_tests, rollover_level the same for the upper end's rollover tests (one-sided);
  • birge_ratio — R = √(χ²/dof) of the fitted points about the line (each in its median's standard deviation from the counts; dof the points less the one slope): the scatter the counts explain is R ≤ 1; when R > 1, K's standard error is multiplied by R (the Birge ratio: never shrunk);
  • points_offered — the points before the range search (the result's n is the points fitted).
source
VisualGeometryToolkitCore.defect_onsets — Method
defect_onsets(masks; source::SensorSource.T = SensorSource.measured) -> DefectList

ONE DefectList from per-exposure defect masks: masks is a vector of exposure => mask (a time quantity and a Bool matrix over the visible raw, master_dark's defects with the shortest exposure of its band, say), all masks of one size. Each photosite flagged in any mask is listed once, its onset the shortest exposure whose mask flags it (the resolver masks it at every exposure from there on: a site flagged at a short exposure and missed at a longer one is a defect at both). The sites are in column-major order of the visible raw. An empty vector or masks of different sizes are an ArgumentError.

source
VisualGeometryToolkitCore.master_dark — Method
master_dark(frames::AbstractVector{<:RawMosaic}; exposure_band, false_defects = 0.01)
    -> (; dark::Dark, read_noise::ReadNoise, defects::BitMatrix)

THE dark-frame estimator: the lens-capped frames of one body at one ISO and one exposure band (whole mosaics of one size, at least 2) into the components of a characterisation's entry, all SensorSource.measured:

  • dark — the Dark: per photosite the temporal mean (the offset: black, DSNU and dark current) and sample variance, one KernelAbstractions work item each, on the frames' device; its channel black (dark.black) a ChannelBlack, per colour index (LibRaw's 0–3) the mean offset of its photosites (defects left out), with its standard error from their spread and count;
  • read_noise — a ReadNoise: per colour index, the root mean temporal variance (the read noise with the rounding, and at a long exposure the dark current's shot noise), its standard error from the χ² spread of the frame count and the photosite count (√(2/(n − 1)/N)·σ² for the mean variance, by first order);
  • defects — a BitMatrix, the photosites whose offset leaves their channel's median by more than z robust standard deviations (the median absolute deviation over the normal's, mad_scale's rule for codimension 1, about the channel's median), z the normal quantile at which false_defects defect-free photosites are expected to be flagged over the whole sensor of N photosites (1 − false_defects / (2N), two-sided, for offsets that are Gaussian about their channel's black). The masks of several exposures make one DefectList by defect_onsets.

A colour index the pattern lacks (a pattern with one green index) has black and read noise 0 (no photosite reads it).

exposure_band is (shortest, longest), time quantities, the exposures the dark serves. A frame of another size, or fewer than 2, is an ArgumentError; a channel without a usable photosite throws NoiseEstimateFailed.

source
VisualGeometryToolkitCore.photon_transfer — Method
photon_transfer(groups::AbstractDict{<:Quantity, <:AbstractVector{<:RawMosaic}}, darks::Darks;
                defects::DefectComponent = NoDefects(), block = 8, normalize = true, significance = 0.001)
    -> FitResult{PoissonGaussian, PhotonTransferMetadata}
photon_transfer(groups::AbstractDict{<:Quantity, <:AbstractVector{<:RawMosaic}},
                darks::AbstractDict{<:Quantity, <:ReadNoise}; sensor_model::SensorModel = SensorModel(),
                block = 8, normalize = true, significance = 0.001) -> FitResult{PoissonGaussian, PhotonTransferMetadata}
photon_transfer(mosaic::RawMosaic; sensor_model::SensorModel = SensorModel(), block = 8, significance = 0.001)
    -> PoissonGaussian

THE estimator of a sensor's Poisson–Gaussian noise for a characterisation's entry, built offline (owner ruling D1: the resolver sensor_model never estimates): the conversion gain K (electrons per code), by a photon-transfer fit of variance against mean, v = σ₀² + μ/K.

The photon-transfer series (groups, exposure => the frames at that exposure, at least 2 each, of one body at one ISO), a MEASURED PoissonGaussian (SensorSource.measured, a dedicated capture) in a FitResult (its estimate), its metadata a PhotonTransferMetadata: per green photosite its TEMPORAL variance over the group's frames, which cancels the scene and every fixed pattern, about its mean μ. Pairs are the case of 2 frames. With normalize (the default) each frame is first scaled to its group's mean (its greens' black-subtracted mean over the group's: a source that flickers between frames adds no μ² term). Per block, the photosites' means and their pooled variance give a point, v̂/v following χ²_{p(N−1)}/(p(N − 1)) over a whole block of p photosites and N frames; every group gives its first N frames, N the smallest group's, so every block follows that one law. The darks come in two forms:

  • a Darks (Core 0.10.4; master_dark's Darks, one per exposure band): each group is served by THE Dark whose band holds its exposure (the Darks' own rule, as sensor_model resolves a frame: an exposure of any number type, 10 * ms and 0.01 s alike). Each group is linearised with its own Dark — its offset map per photosite and its channel black — and its intercept σ₀²(t) is that Dark's mean green variance over the photosites defects leaves (cut to the group's exposure: a site is masked from its onset); σr's standard error is carried from the Dark's frame and photosite counts, √(2/((n − 1)N))·σ₀² (masterdark's rule). A group whose exposure no band serves is an ArgumentError: never the shared black, never another band's Dark;

  • a Dict of ReadNoise (exposure => masterdark's ReadNoise over the darks of that exposure; scheduled for removal in the next breaking wave, use the Darks form): every group linearised with `sensormodel's black and defects (ONE model for every exposure), its intercept the greens'σ[c + 1]²ofdarks[t]. A group whose exposure has no entry indarks(EQUAL in seconds:1//100 sis not theFloat640.01 s) is anArgumentError: never interpolated, never substituted. σ_r's standard error is the ReadNoise's (0` when it states none).

  • THE INTERCEPT is fixed at each group's own exposure t; the fit is the slope alone, v − σ₀²(t) = μ/K.

  • THE RANGE is derived from the data, at least 3 points: the upper end below the variance's rollover (the top point dropped while it lies significantly below the line fitted to the points under it, one-sided), then the lower end, the lowest point from which the μ² test on the line succeeds (the lowest point dropped until it does). Each search runs its tests at significance / m, m the tests it can perform (Bonferroni: the search's family-wise false-alarm rate is significance); the range (in codes and as fractions of the white level), m and the levels are in the result's metadata.

  • THE STANDARD ERROR of K is the fitted line's from the counts, multiplied by the Birge ratio R = √(χ²/dof) of the fitted points when R > 1 (the points scatter more than the counts explain); when R ≤ 1 it is unchanged, never shrunk. R is in the result's metadata.

  • The result's σr is √σ₀² of the fitted points' darks (their median), its standard error the darks' at that median (the mean of the two middle points' when it falls between two exposures), by first order; a characterisation states σr per colour from master_dark's ReadNoise.

  • The result's return code and stop rule are the weighted line's: Success and IncrementSize when its weights reached their fixed point (iterations the reweightings), MaxIters and IterationCap when they did not.

One frame (mosaic), in the line of Foi et al. (2008), an ESTIMATED PoissonGaussian (SensorSource.estimated): unreliable on flats, scheduled for removal in the next breaking wave; use the multi-frame method. A single flat frame cannot certify its own K (its brightness range gives the μ² test little power); for flat fields use the multi-frame photon-transfer method. Each 2×2 cell's two greens see the same light, so their difference d carries no scene where the scene is flat across the cell, and Var d = 2(μ/K + σ_r²) about their black-subtracted mean μ. Per block, the cells' mean and the difference's sample variance give a point (μ̂, v̂ = s²/2), v̂/v following χ²_{n−1}/(n − 1) over a whole block of n usable samples (cells); the line's intercept σr² is fitted, over every point, with its standard error (σr stated for every colour index, the greens'). A fixed pattern between the two greens (their gain or response mismatch) does NOT cancel in d: it adds a term in μ², which the μ² test REFUSES; a frame that gives fewer than 4 points, on which the μ² test cannot run, is refused too (its K is never returned untested).

Both: per block of block × block cells (one KernelAbstractions work item each, on the frames' device), a sample with a photosite at the white level, or under the model's defect mask, is left out (CLIPPED, never modelled), and only whole blocks (the most usable samples any block has) enter the fit. The blocks are sorted by μ̂ into points of equal count (min(32, blocks ÷ 8); the series': per exposure, so a point never mixes two exposures' intercepts, an exposure of fewer than 8 whole blocks left out); each point's median μ̂ and median v̂ / q (q that law's median, so the ratio is median-unbiased for v). The line is fitted by weighted least squares, each point weighted by the inverse of its median's variance from the counts, v²/(4 m f(q)² q²) over a point of m blocks (f the law's density: the sample median's asymptotic variance), with v the line's own prediction (iterated to a fixed point from a start weighted at each point's own value, at most 100 reweightings). The median is robust to the blocks a scene edge raises. The standard errors are the fitted line's covariance from those counts; K's by first order.

THE μ² TEST: the points fitted again with a μ² term, v = … + μ/K + cμ², at the same weights, and c significant when |ĉ / se(ĉ)| exceeds the two-sided normal quantile at the test's level (one frame: significance, 3.29 at the default 0.001, over every point; the series: the lower end's search). It never returns a K biased by a μ² term it detects. The default significance is configs/default.toml's [photon_transfer] significance.

sensor_model gives what is subtracted and left out, as it does for LinearRaw: its black slot's per-colour black (a Dark's offset per photosite in its place) and its defect slot's photosites; its noise slot is not read. Throws NoiseEstimateFailed with the reason when fewer than 16 whole blocks of at least 8 usable samples remain (a saturated frame), the points hold one level only (no slope), one frame gives fewer than 4 points, one frame's μ² term is significant, one frame's weighted line does not converge, the series' μ² test fails on every contiguous range of at least 3 points, or the fit is not physical (K, or one frame's σ_r², not positive). A group of fewer than 2 frames, frames of another size, pattern or white level, or a group's exposure without darks, is an ArgumentError; dark maps that do not cover the frames a DimensionMismatch.

source
VisualGeometryToolkitCore.SensorSource — Module
SensorSource

Where a sensor component's values came from, SensorSource.T (an isbits @enum): SensorSource.measured (a record of dark frames, flats or a photon-transfer capture), SensorSource.estimated (fitted by an estimator, photon_transfer, with its standard error), SensorSource.specified (stated by a person: a datasheet, a config, TOML literals) or SensorSource.file (the RAW file's own metadata: FileBlack's). string(s) is its tag ("measured", …). Each present component holds one in its source field. A specified component wins over a measured one at the same slot (sensor_model).

source