Lutra's RAW pipeline

“Clutter and confusion are failures of design, not attributes of information.”

A camera RAW file is not a picture. It is a measurement — millions of single numbers, one per light sensor on the camera's chip, plus a sheet of notes about how those numbers should be interpreted. Turning it into the photograph you see requires a pipeline of computations — some straightforward arithmetic, some educated guessing — and every program that does this (Lightroom, your phone, Lutra) makes choices along the way. This document explains what those numbers are, what the pipeline does to them, and exactly which choices Lutra makes and why. It assumes you know what a pixel is and nothing else; the terms the rest of the document leans on are defined up front in §0.

0 · One number per pixel

A digital camera's image sensor is a grid of millions of tiny light meters called photosites. Each photosite records exactly one thing: how much light fell on it. Not what color. Not where the light came from. Just an intensity — and it records it as a plain number, proportional to the light (twice the photons, twice the number), typically with 12–14 bits of precision (4,096–16,384 possible levels per site, versus the 256 per channel a JPEG stores).

Color is added later, and not by the sensor: the photosites are covered by a grid of tiny colored filters — the color filter array, or mosaic — so that each photosite sees light through exactly one red, green, or blue filter. The classic pattern is Bayer: 2×2 blocks with two greens, one red, one blue, named for Bryce Bayer who invented it at Kodak. The extra green is a nod to your eyes, which are most sensitive to green light; sampling it at double density gets the most detail out of the scene.

Because each photosite sees only one color, a RAW file's sensor data is a mosaic: 75% of the numbers are “missing” two color channels each. The file is also just numbers plus notes — the sensor readings, losslessly compressed, accompanied by a metadata section recording everything the decode needs: the mosaic pattern, the black level (what the sensor reports for darkness), the white-balance multipliers the camera chose, the camera's color calibration matrices, the orientation, and — usefully — a small JPEG the camera rendered as a preview.

1 · Why the file is not a photo yet

When you shoot JPEG, the camera's firmware runs the exact pipeline this document describes inside the camera, in a few milliseconds, with the camera's own taste: a white balance it guessed, a contrast curve it likes, sharpening, noise reduction, and lossy compression. The result is a finished, viewable image — and everything else is gone. The mosaic, the extra precision, the choice of interpretation: deleted, permanently.

RAW shooting keeps the measurement. The file is the negative: uninterpreted, unprocessed, and — because the sensor's numbers are linear and 12–14 bits deep — holding far more information than any 8-bit rendering of it. Expose for the highlights and the shadows still contain recoverable detail; pick a different white balance in post and the color is re-derived from data that was always there, rather than re-painted over a baked guess. Every serious photo workflow converges on the same shape: keep the negative, interpret it later.

This is also why decoding is a program, not a file conversion. A RAW file has no “picture” inside it (the embedded JPEG aside — see §6); it has measurements and notes. The program's job is to interpret the notes and turn the measurements into pixels — and the interpretation is where the choices live.

2 · The mosaic becomes a picture: demosaicing

Every pixel of the final image needs a red, a green, and a blue value. The sensor only measured one of the three at each site. So the missing two must be reconstructed from neighboring sites — a step called demosaicing (or de-Bayering), and it is the one genuinely opinionated step in the pipeline. A trivial demosaicer looks at a red-filtered site, takes its red measurement, and copies the green and blue values from the nearest neighbors. That works but produces soft, jagged, color-speckled results at fine detail. A good demosaicer first looks at the structure of the neighborhood — where the edges are, which way they run — and only then interpolates along the edges rather than across them, which keeps edges crisp and kills most false color.

The algorithm Lutra uses, AHD (adaptive homogeneity-directed demosaicing), is exactly that kind of edge-aware reconstruction: it produces two candidate guesses for each pixel (one from horizontal structure, one from vertical), and picks the better one by comparing which guess makes a smoother neighborhood. It comes from the dcraw lineage and is the standard mid-tier choice: dramatically better than naive interpolation, a bit slower, and — relevantly for §7 — the best algorithm this build is allowed to ship.

Two facts keep demosaicing from being as lossy as it sounds. First, real scenes are smooth at the pixel scale; neighboring photosites almost always recorded nearly the same light, so the guess is usually excellent. Second, the mosaic samples green at double density precisely because green carries most of the perceived detail — the colors being reconstructed from neighbors (red and blue) are the ones your eyes resolve coarsest anyway. The artifacts that remain — zippering on high-contrast diagonal edges, faint false color in fine repeating patterns like fabric — are the same artifacts every RAW developer lives with, and they are the reason demosaic quality is a thing at all.

3 · The decode pipeline, in seven steps

Here is the whole journey from file bytes to displayable image, in one picture — and, in miniature, what the camera's firmware does when you shoot JPEG:

In order of appearance:

1. Unpack. Sensor data is compressed per-camera-format (losslessly, like a zip file for measurements — though Canon's C-RAW and Nikon's HE/HE* are lossy shortcuts, §8). Unpacking is the mechanical step: get the numbers back out.
2. Black / white levels. A photosite with no light does not report zero — electronics leak, so each channel has a black level (a stored offset, e.g. 64). The decode subtracts it, then scales the whole range so the sensor's white level maps to 1.0. Skip this and blacks come out tinted or lifted, and every channel is miscalibrated. The levels come from the file's metadata — never guessed.The black level is per-channel, which is why skipping this step shows up as a color cast in the shadows specifically — each channel's offset is different. It is the classic first thing to go wrong in a hand-rolled RAW decoder.
3. White balance. Light has a color. Tungsten light is orange; a gray card under it records orange-gray until each channel is scaled to cancel the tint. The camera measures (or guesses) the right scaling at the moment of shooting and records it in the file as multipliers — the as-shot white balance. The decode applies those multipliers, so a white wall comes out white. (§4 is devoted to this step, because it is the one with a real design decision in it.)
4. Demosaic. The mosaic becomes full color (§2). This is the step that actually invents information, and the only one where the algorithm choice is visible in the output.
5. Color matrix. The sensor's red, green, and blue filters are not the same colors as your screen's red, green, and blue primaries — every camera's spectral response is its own. The decode converts camera color space to a standard one with a 3×3 matrix recorded in the file's calibration data (§5).
6. Defects & orientation. Dead and hot pixels (photosites that always read dark or always read bright) are detected and repaired from their neighbors, automatically. The EXIF orientation is applied — some cameras store the image sideways and a flag says which way is up.
7. Tone encode. The pipeline so far has been working in linear light — numbers proportional to photons. Screens and image files don't store that; they store sRGB, a curve that packs the brightness range our eyes actually care about into 8 bits (the same curve architecture.html §4 discusses at length). The final step applies that curve, producing the sRGB-encoded image the rest of the app treats as a normal photo.

Steps 3–7 have a musical-chairs quality: the camera firmware performs them too, at the moment of shooting, with its own taste, and then throws away the measurements. A RAW decoder performs them later, with your taste, on the measurements that were kept.

4 · White balance, and the “as-shot” decision

White balance is the step with a real design decision in it, so it deserves its own section. The physics: light has a color temperature. A gray card under warm tungsten light reflects warm light — it records as orange. To make it record as gray, the red channel must be scaled down and the blue up, by amounts the camera works out from its metering. Those multipliers are written into the RAW file: the as-shot white balance.

The alternative is a neutral decode: apply no white balance at all, or estimate one (gray-world), and let the user's white-balance controls own the color entirely. Both are defensible — the decode is a moment of interpretation, after all. Lutra applies as-shot, and the reason is the White Balance adjustment layer's range. That layer scales the red and blue channels by at most ±30% at the slider extremes. As a trim on top of as-shot — the slider's zero meaning “the camera's recorded white balance” — ±30% is plenty. As a correction on top of a neutral decode it would be hopeless: under tungsten, the correct blue gain can be 2.5× — far beyond any slider. A neutral decode plus a bounded slider cannot even reach correct white balance; an as-shot decode plus the same slider lands the user exactly where the camera preview did, and lets them trim from there.

5 · From camera color to screen color

Step 5 deserves one section of its own because its failure mode is the most confusing. Camera sensors do not see color the way screens display it: the filter dyes are different, the sensitivities overlap differently. The same red apple records as one triplet of numbers on a Canon sensor and a different triplet on a Sony sensor. To make every camera's files come out as the same colors, each camera's calibration records a 3×3 color matrix — usually two of them, for two reference light sources, interpolated by the light the file was shot under — and the decode multiplies every pixel by it.

If the matrix is skipped, the image is not wildly wrong; it is plausibly, insultingly wrong — colors that are recognizable but desaturated and shifted, skin tones that look muddy, the classic “washed-out RAW” complaint. The fix is not a filter; it is the matrix that was in the file all along. Lutra applies it always (useCameraMatrix: 3, which means “always,” not “only when a white balance is applied”). There is no creative upside to skipping it — the artistic control lives in the layers, not in decode-level omission.

6 · What Lutra does when you open a RAW file

Everything above is the general science. Here is the specific machinery. When you pick a RAW file, Lutra routes it by sniffing the file's magic bytes and extension — a RAW file is, structurally, a TIFF-family container, so the sniff is cheap and reliable. The flow:

Three things make this flow worth explaining:

The embedded JPEG preview is the instant half. Every RAW file contains a JPEG the camera rendered at the moment of shooting (§1). Extracting it is nearly free (tens of milliseconds), so it fills the gallery tile and the editor's loading placeholder immediately — the editor never stares at a blank screen while decoding. The full decode replaces it in under a couple of seconds on a desktop. The embedded JPEG is not the same image as the decode: it carries the camera's tone curve and processing, so it is strictly a placeholder, never a substitute (§6.3 shows the one place that distinction bites).

The decode runs in a worker. A 24-megapixel RAW takes on the order of a second to decode on desktop hardware — long enough that doing it on the UI thread would freeze the whole app. It runs in a Web Worker instead, with the decode split into coarse stages (unpack, white balance, demosaic, color conversion) and each stage reported back as progress. The main thread stays responsive; the progress bar is honest, and the embedded JPEG fills the wait.

The output is a 16-bit sRGB image. Here is the decision that makes the whole pipeline drop into place: the decode runs steps 2–7 of §3 — including the tone encode — and emits sRGB-encoded, 16-bit pixels. That is the same encoding a JPEG file stores, just with more precision per channel (the engine's existing pass-0 srgbToLinear inverts the curve exactly — architecture.html §4). The GPU upload pads the three color channels to four (WebGPU has no RGB16 format) and writes an rgba16unorm texture. The engine never knows the difference between this and a JPEG: same colorspace contract, same pipeline, just 16 bits of precision instead of 8. There is no linear-decoding path, no engine flag, no second chain — the complexity lives entirely in the decoder, where it belongs.

6.1 · The previews: tiles, placeholders, and the LUT bar

The one place the embedded JPEG's camera-rendering shows through is the LUT bar — the filmstrip of “your photo under every LUT.” Its thumbnails are generated from a small preview of the user's photo (LUT.html §5, docs/adr/0013), and for RAW files that preview must come from the decode, not the embedded JPEG: the embedded JPEG carries the camera's tone curve (typically brighter), while the decode is deliberately un-brightened (noAutoBright — the camera's recorded levels, used as-is, so the Exposure layer owns brightness). A LUT thumb based on the embedded JPEG would show the film look on a visibly brighter image than the editor actually renders — the bar's whole promise is “see it on your photo, as it will look.” So the decode worker derives a small (~1024×1024) sRGB preview from the full decode itself, and the filmstrip consumes that. Gallery tiles and the loading placeholder, which are placeholders by nature, use the embedded JPEG directly.

6.2 · The engine contract, in one sentence

A RAW file decodes to the same thing a JPEG file is — an sRGB-encoded image — at higher precision. Everything downstream, from the chain assembler to compare mode (which binds the decoded source texture) to export, treats it as an ordinary source image. That is the entire integration story, and it is why this document fits in the series: the RAW pipeline is a source pipeline; the grading pipeline is documented in architecture.html.

6.3 · Memory, and the file's second life

A 24-megapixel decode occupies roughly 192 MB as RGBA16 pixels in the worker's memory before upload, and the worker's wasm heap grows from 256 MB to accommodate it — comfortably within the browser's ~2 GB wasm ceiling up to very large sensors (~100 MP is the practical cap). Closing the image disposes the worker and releases the heap. The original file bytes also live on, untouched, as the Edit's source — an Edit row is now commonly 25–100 MB, which IndexedDB handles fine (a storage-persistence request at first save guards against eviction; per-edit size has always been part of the Edit summary, context.md).

7 · The decoder: LibRaw in your browser

The decode itself is performed by LibRaw, the open-source C++ library at the heart of most RAW software — it understands the sensor formats and compression schemes of essentially every camera that makes RAW files, including Fuji's X-Trans mosaics and the newest bodies (as long as the library version is recent enough). Browsers cannot run C++ directly, so LibRaw is compiled to WebAssembly — a low-level binary format browsers execute at near-native speed — and bundled with the app (~1.4 MB, built with SIMD and multithreading enabled). The wasm build, the thin C++ wrapper around it, and the TypeScript client are ours: a fork of the LibRaw-Wasm project, rebuilt from source by bun run build:raw (docs/adr/0014).

Owning the build is a deliberate choice, and it buys three concrete things:

• Progress. The stock wrapper reports nothing during a decode; ours wires LibRaw's progress callback so the editor can show the stage-based bar of §6.
• Version freedom. Cameras ship new formats on their own schedule (Canon C-RAW, Nikon HE*, new bodies). The build pins LibRaw 0.22.1 and can be bumped and rebuilt whenever — a rebuild, not a rewrite.
• Trust. The shipped wasm binary is built from pinned, inspectable sources rather than consumed from a third party.

One boundary is worth naming plainly. LibRaw's best demosaic algorithms (AMaZE, DCB, and friends) are licensed GPL, and including them would drag the whole app's licensing posture along. The build therefore stops at AHD (§2) — the best algorithm in the permissively-licensed core. AHD is a perfectly respectable quality tier; it is simply not the absolute ceiling, and the ceiling is a licensing decision, not a technical one.

8 · The formats

“RAW” is a family, not a format. Every maker has its own container and its own compression; the decode library's job is to know them all. The ones Lutra accepts:

Two special cases are worth understanding, because they invert the “RAW = mosaic” picture. A linear DNG is a DNG that stores already-demosaiced RGB — three values per pixel, no mosaic — but still linear (un-tonemapped) and still carrying the calibration metadata. iPhone ProRAW (12 MP) is exactly this: the file is already a “photo,” missing only the color matrix and tone curve. Lutra gives linear DNGs no special treatment: if LibRaw decodes them (it does, for dcraw-style pipelines), they work; if a file fails, the failure is an honest, readable error. The 48 MP ProRAW variant, by contrast, is a normal Bayer DNG.

And the embedded JPEG preview (§6) is technically a second file inside the first — which is why RAW files are “two things at once”: a measurement waiting for interpretation, and a finished picture the camera already made. Lutra shows you the second instantly, then does the first properly.

9 · Reference tables

The decode configuration, once (these are the actual settings the decoder is opened with — docs/adr/0014):

useCameraWb: true        // as-shot white balance (§4) — the untouched image
                         // matches what the camera previewed
useCameraMatrix: 3       // always apply the camera color matrix (§5)
outputColor: 1           // sRGB primaries
outputBps: 16            // 16 bits per channel
gamm: [1/2.4, 12.92]     // the sRGB curve — inverted exactly by the engine's
                         // pass-0 srgbToLinear (architecture.html §4)
userQual: 3              // AHD demosaic (§2) — the ceiling of this build
highlight: 2             // blend: blown highlights reconstruct from neighbors
                         // instead of clipping to a tinted white
noAutoBright: true       // no histogram brightening — the camera's recorded
                         // levels, as-is; the Exposure layer owns brightness
userFlip: -1             // apply the EXIF orientation
halfSize: false          // full resolution, always