Notebook

3D representations, from Zwicker to Gaussian Splatting

Interactive notes to understand (and remember) how 3DGS works: every lesson has manipulable figures — move the sliders, drag things around, train a model in the browser.

How to use — serve the folder over HTTP (the demos use ES modules from CDN): python -m http.server 8010 from the root, then open http://localhost:8010/. An internet connection is required (Three.js and KaTeX come from CDN).

Lessons

00 Prerequisites: eigenvectors, eigenvalues, the Jacobian The mental images before the math: rails and trains, rolling pins, local maps, and the determinant as an area factor.
01 The primitive: the anisotropic 3D Gaussian μ, Σ = R·S·Sᵀ·Rᵀ, opacity, spherical harmonics — play with all 59 degrees of freedom.
02 'Traditional' splatting: kernels, not parameters QSplat / EWA surface splatting: the Gaussian as a reconstruction filter. Points-with-holes vs splats.
03 The forward pass: volumetric compositing Transmittance along a ray, the same equation as NeRF, Lagrangian vs Eulerian view.
04 Training: the scene is the model Analysis-by-synthesis: train 2D gaussian splatting live in the browser, with densification.
05 The EWA projection: Σ′ = J·W·Σ·Wᵀ·Jᵀ (coming) why a Gaussian stays ~Gaussian on screen.
06 Compression and streaming: SPZ, SOG, LoD (coming) quantizing a mixture; Spark 2.0's coarse-to-fine.

Experiments

The research logbook: what we tried on our data, which decisions we made and which paths we abandoned (and why). It is the empirical complement to the lessons — and the shared memory for doing research together: before repeating an attempt, check here first.

E001 Photorealistic 3DGS from our data: how much, where, and why not PSNR 28.7 along the capture path, collapse 20 cm off it; the intrinsics recovery; the depth-regularized series (E2/E3) and the scale_reg own goal.

Monitored papers

The watchlist: recent papers that could improve our 3D reconstruction pipeline. One entry per paper — what it does, why it matters to us, and what event would change its status. Discarded papers keep their entry (with the reason), so we never re-evaluate from scratch. Mirrored by an index note in the EC Obsidian vault.

P Papers watchlist — candidates for the reconstruction pipeline Currently watching: P001 PixWorld (unified generation + reconstruction in pixel space, feed-forward pixel-aligned gaussians) — the generative half is the missing tool for E001's capture-corridor wall.

3D Graphics

Outside the splatting thread: "classic" real-time graphics, as a tool to see the rendering pipeline. Interactive Three.js scenes where what matters is not just the result, but how the GPU gets there — opening up the G-buffer and its passes.

G01 A closed room, and the G-buffer inspector A closed environment with three windows, cube/pyramid/sphere and a flying camera (WASD + mouse); on top, a deferred renderer with a real MRT G-buffer and a panel to step through the 11 pipeline stages — with a masking divider to compare each step against the Beauty.

Demo

The best-so-far 3DGS scene, explorable in the browser: the assets come from S3, the viewer (Spark + three.js) runs entirely in the page.

World models

The learning thread: what a world model actually is and how objective, representation, dataset shape, uncertainty, and training loop determine what it can predict. A research map connecting latent control models, video foundation models, interactive simulators, and the planning systems built on top of them.

W01 World models: objectives, data shapes, and training methods Four objective families, five dataset shapes, the major training regimes, an interactive goal-to-design selector, evaluation traps, and the primary-paper reading map.

Research collabs

Working briefs for conversations with researchers whose methods may connect to our world-model data stack. Each note separates the promising conceptual bridge from what still needs to be demonstrated, and ends with a concrete experiment or interview agenda.

RC01 Michele Zhu — from semantic video to a trajectory codec Can semantic communication become a layered video + action format for world-model data? The fit, the gaps, a codec sketch, and the equal-byte experiment that would test it.

Data streaming

The infrastructure thread: how our dataset reaches a world-model lab's GPU without each researcher rebuilding the plumbing. Where the splatting lessons are about representing scenes, these are about moving them — the network stack, the protocols, and why "stream straight to the GPU over TCP" is the one thing you can't do.

S01 The network stack for GPU data streaming Every layer between an S3 object and a tensor in VRAM, a "why this can't stream to the GPU" table across TCP/QUIC/NFS/RDMA/GPUDirect/GDS/cuObject, and why research landed on the SOTA it did. The seamless S3→GPU wedge for world-model training.

Why this notebook

It was born out of the 3d-viewer work (Potree, the flat renderer, and the 3DGS test on the GPU box): these concepts genuinely matter for deciding how to render and stream our reconstructions. The thesis that ties all the lessons together:

Thesis — in traditional gaussian splatting the Gaussian is a reconstruction filter applied to measured geometry; in 3DGS it is a latent parameter of a generative image model, estimated by gradient descent through a differentiable renderer. Same primitive, two epistemologies: forward rendering vs inverse problem.

The manim/ folder: scene scripts to generate video versions of the animations (instructions in the folder's README).