← All visualizations
Interactive note 005 Created with Opus 5

A visual exploration of Gaussian splatting

A scene made of nothing but ellipsoids of coloured fog. Sort them by depth, blend them in order, and every parameter — where they sit, how they lean, how solid they are — falls out of a gradient.

01

The primitive

One splat is five numbers and a colour

A splat is a Gaussian with a colour and an opacity. Its shape never comes from the covariance directly — that would let the optimizer wander into matrices that are not positive semi-definite. It is built instead from a rotation and two scales, which cannot.

drag μ, s₁ or s₂ directly on the canvas

Parameterisation

Rotation and scale, never Σ itself

The two handles are the columns of R, stretched by s₁ and s₂. Whatever you do to them, the matrix below stays a valid covariance: it is a product that cannot come out indefinite.

\[\Sigma = R\,\mathrm{diag}(s_1^2,s_2^2)\,R^{\top}\] \[G(x)=\exp\!\left(-\tfrac12 (x-\mu)^{\top}\Sigma^{-1}(x-\mu)\right)\]
Σ
—— ——
Anisotropy
—
Orientation
—
1σ footprint
—
Peak opacity
—

Squash one axis and the splat becomes a needle that can trace an edge. This anisotropy is the whole reason a few hundred thousand Gaussians can stand in for a scene: each one is allowed to be the wrong shape everywhere except where it is needed.

02

The renderer

Blending is a walk from front to back

There is no ray marching and no network here. Each pixel visits the splats that cover it in depth order, spends a little of its remaining transmittance on each one, and stops when nothing is left. That is the entire renderer.

transmittance left at the probe — covered —

One pixel, opened up

Each splat spends what is left

\[a_i=\alpha_i\,G_i(x),\qquad T_i=\prod_{j<i}\left(1-a_j\right)\] \[C(x)=\sum_{i=1}^{n} c_i\,a_i\,T_i \;+\; T_{n+1}\,c_{\text{bg}}\]
#GaTa·T
    resulting pixel

    Switch to arbitrary order and the picture changes although not one splat moved. Alpha compositing does not commute, which is why a real renderer sorts every splat by view depth for every frame — and why the sort, not the maths, is the expensive part.

    Drag the numbered splats over each other. The list above is recomputed for the pixel under the crosshair — click anywhere on the canvas to move it.

    03

    The optimizer

    Where the splats come from

    Nobody places them. A few hundred random Gaussians are dropped on the image and every one of the nine numbers describing them is pushed downhill by the same gradient that flows back through the blend above. This runs live in your browser — analytic gradients, Adam, and the clone / split / prune rule that decides how many splats the picture deserves. Or open the sandbox and paint your own target: the optimizer never stops, so the population reorganises under the brush.

    iteration 0 PSNR —
    gradient descent on 96 × 96 pixels

    The backward pass

    Nine gradients per splat

    The loss is plain ℓ₂ between render and target. Because the renderer is a product of transmittances, the derivative for splat i needs everything composited behind it — so the backward pass walks the same list in reverse, rebuilding each Tᵢ by division.

    \[\frac{\partial \mathcal L}{\partial a_i} = \nabla_{C}\mathcal L \cdot \left(c_i T_i - \frac{S_i}{1-a_i}\right),\qquad S_i=\sum_{j>i} c_j a_j T_j\] \[\frac{\partial \mathcal L}{\partial \Sigma} = -\Sigma^{-1} \frac{\partial \mathcal L}{\partial \Sigma^{-1}} \Sigma^{-1},\qquad \frac{\partial \mathcal L}{\partial W}=2\,\frac{\partial \mathcal L}{\partial \Sigma}W, \quad W=RS\]

    Every 60 steps the splats with the largest positional gradient are duplicated: small ones are cloned into the gap they are being pulled towards, large ones are split in two with their scale divided by 1.6. Splats that fade below 2 % opacity are deleted. Detail appears where the loss asks for it — watch the dashed count climb on the hard-edged target.

    splats — parameters — vs. pixels — ℓ₂ loss —
    04

    The projection

    Lifting the ellipse into a camera

    A perspective camera is not a linear map, so a projected Gaussian is not exactly a Gaussian. EWA splatting linearises the projection at each splat's centre and keeps the quadratic form — one Jacobian per splat, one 2×2 covariance in pixels, and the renderer from chapter 02 does the rest. Everything below is rasterised on the CPU, one pixel at a time.

    — Gaussians — after culling —

    drag to orbit

    EWA splatting

    Σ′ = J W Σ Wᵀ Jᵀ

    W rotates the covariance into camera space; J is the Jacobian of the perspective divide evaluated at the splat centre. Drop the third row and column and a 3-D ellipsoid has become a 2-D ellipse in pixels.

    \[J=\begin{bmatrix} f/t_z & 0 & -f\,t_x/t_z^{2}\\[2pt] 0 & f/t_z & -f\,t_y/t_z^{2}\end{bmatrix}, \qquad \Sigma' = J W \Sigma W^{\top} J^{\top}\] \[c_i(d) = c_i^{(0)} + \langle a_i, d\rangle \quad (\ell = 1 \text{ spherical harmonic})\]

    Turn off the depth sort — the splats then arrive in the arbitrary order they sit in memory — and the far side of the surface starts breaking through the near side in speckles: with opaque splats the order is the occlusion. Turn off the low-pass filter and distant splats thin below one pixel and start to shimmer; the reference implementation simply adds 0.3 px² to the diagonal of Σ′ so that nothing is ever narrower than a sample.

    rasterised — buffer —