Streaming S01
The network stack for GPU data streaming
What every layer between an S3 object and a tensor in VRAM actually does — and why "just stream it straight to the GPU over TCP" is the one thing you cannot do. A map of why research landed on the SOTA it did.
1 · The thing people picture vs. the thing that exists
The mental model behind "stream the dataset directly to the GPU" is a single wire from a bucket to VRAM. That wire does not exist over the open internet. A byte leaving an S3 object has to cross four independent gaps, and each gap is governed by a different technology with its own limits. The illusion of "direct" is really about which of these gaps you manage to skip.
Every streaming system in existence — HF datasets, MosaicML, WebDataset, DALI, the
research SOTA — is a specific set of answers to: which gaps do I cross in software on the CPU, and
which do I hand to dedicated hardware that lets the CPU step out of the way? "Direct to GPU" in the
literature always means one precise thing: skipping the Host (CPU+RAM)
bounce buffer so bytes land in VRAM without the CPU copying them. That skip is real — but only over the
technologies in the green rows below, none of which is the public internet.
2 · The transport layer: why not "just TCP"?
When I said earlier we "cannot use TCP", this is what I meant — not that TCP fails to move bytes (it moves nearly all of them today), but that TCP structurally cannot deliver a byte into GPU memory without the CPU. TCP is a kernel, CPU-driven protocol: every segment is reassembled, checksummed and copied by the host CPU into host RAM. The GPU is never a party to a TCP connection. So a TCP-based path always pays the host bounce-buffer and a per-byte CPU tax. The SOTA is a family of protocols that were designed specifically to remove the CPU from the data path.
| Transport / protocol | What it is | Why it can — or can't — stream into the GPU | Verdict |
|---|---|---|---|
| TCP/IP HTTP · S3 API |
The default reliable transport. Everything on the public internet, including every S3 GET, rides it. |
Kernel-driven: the CPU reassembles, checksums and copies every segment into host RAM. The GPU can't terminate a TCP flow, so a copy across PCIe is mandatory afterwards. It's the bounce buffer, by design. | No — CPU in the loop |
| QUIC / HTTP/3 | Modern UDP-based transport (lower head-of-line blocking, faster handshakes). What a fast HTTP object client uses today. | Better latency, same fundamental problem: it runs in user/kernel space on the CPU. Congestion control, loss recovery and crypto are all CPU work; the payload still lands in host RAM first. | No — still host-terminated |
| NFS / POSIX mount "S3 as a filesystem" |
Mounting a bucket (s3fs, Mountpoint, goofys) so training code sees files. Convenient, and what most naive pipelines reach for. | Every read becomes an HTTP range request under a filesystem shim, then goes through the page cache on the CPU. Throughput measured at ~300 MB/s vs 10+ GB/s local NVMe — ~30× slower — and never touches the GPU directly. | No — slow & CPU-bound |
| RDMA InfiniBand · RoCEv2 · iWARP |
Remote Direct Memory Access. The NIC writes data straight into a pre-registered memory region on the remote host, bypassing its CPU and kernel entirely. | This is the enabling primitive. Combined with GPUDirect, the NIC's DMA engine can target GPU VRAM directly — the CPU never touches the payload. But RDMA needs a lossless, low-latency fabric: it lives inside a datacenter, not across the WAN. | Yes — the foundation |
| GPUDirect RDMA GPU ↔ NIC |
NVIDIA tech letting a third-party NIC read/write GPU memory over PCIe without staging in host RAM. Pairs with RDMA above. | Removes the last copy: remote data → NIC → GPU VRAM, CPU out of the path. This is exactly the "direct to the GPU" people imagine — but it requires the sender, the fabric, and the NIC to all speak RDMA. | Yes — datacenter only |
| GPUDirect Storage (GDS) cuFile |
A DMA path from storage (local NVMe, or NVMe-oF over a fabric) into GPU memory, skipping the CPU bounce buffer. Exposed via the cuFile API. |
Real and shipping — but only for block/file storage on RDMA fabrics (NVMe, NVMe-oF, Lustre, WekaFS, GPFS). It does not speak S3/HTTP. If prerequisites aren't met, cuFile silently falls back to a CPU bounce buffer ("compatibility mode"). |
Partial — not object storage |
| cuObject / GPUDirect-for-S3 tech preview |
NVIDIA's newest piece: object-storage semantics (S3-style) with an RDMA data plane straight into GPU memory. The control plane is HTTP; the data plane is RDMA. | The closest thing to "S3 direct to GPU" that exists — but the data plane still needs RDMA (InfiniBand/RoCEv2) between a compatible storage server and the GPU box. As of early 2026 it was a technical preview, not general availability, and confined to datacenter fabrics. | Partial — emerging, LAN-only |
| NCCL / NVLink GPU ↔ GPU |
The collective-communication layer for multi-GPU training (all-reduce, broadcast). NVLink/NVSwitch is the on-node GPU-to-GPU fabric. | Not an ingest transport — it moves gradients and activations between GPUs already holding data, not data from storage. Relevant because once one rank has a batch, sharing it GPU-to-GPU is cheap; getting the first copy in is the hard part this page is about. | N/A — intra-training, not ingest |
3 · So what is actually reachable over the WAN?
This is the pragmatic core for us. True GPU-direct ingest is a datacenter-fabric capability, and world-model labs training in the cloud (AWS, GCP, CoreWeave, Lambda) mostly are inside such fabrics — but the object store they pull from usually is not on the same RDMA fabric as their GPUs. So the realistic architecture is a relay, not a single wire:
The "seamless" experience we can honestly promise is: the researcher never runs a download step and their local disk never fills up — not "bytes teleport from our bucket into VRAM." The WAN hop is hidden by concurrency and prefetch (the well-known result: a naive loader leaves the GPU 95% idle on S3, but async/threaded prefetch recovers most of it, reaching ~67% of local-disk throughput). The GPU-direct part happens on the last hop, from an in-region cache the SDK manages, where a real RDMA/GDS fabric may exist.
| Layer | Bottleneck it removes | Technique the SOTA uses |
|---|---|---|
| WAN latency | GPU waiting on round-trips | Deep prefetch queues, hundreds of concurrent range requests, GIL-free threading (Meta SPDL) |
| Egress cost & re-reads | Paying per-GB every epoch | Co-locate storage in-region; zero-egress stores (Cloudflare R2); in-region NVMe cache warmed once |
| Random-access shuffle | HTTP hates random reads | Shard + shuffle shards, then a bounded in-memory buffer; global shuffle done offline (the honest limit) |
| Host-RAM bounce buffer | CPU copying every byte | GDS / cuFile from the local cache; RDMA fabric where available |
| Decode (esp. video) | CPU decoding frames | Hardware decode on the GPU — NVDEC for video, nvJPEG for images (DALI) |
4 · The modality that changes the math: video
For text and images the bytes are small and the incumbents are mature. Video is different, and it is our vertical. A world-model dataset is enormous, and the expensive step is not the network but the decode: turning compressed H.264/H.265 into frame tensors. This is where "to the GPU" stops being marketing and becomes real silicon — the NVDEC hardware decoder on the GPU turns the video bitstream into frames without the CPU, right next to where training needs them.
5 · What this means for the "ec" streaming layer
We are not going to build a new transport — RDMA, GPUDirect and GDS already exist and NVIDIA owns them. Our seamless layer composes them and hides the relay. Concretely, the defensible product is the orchestration the table above implies, packaged so a lab writes three lines instead of building this:
- WAN → cache → GPU relay, managed for the user: prefetch, in-region NVMe cache, and GDS/NVDEC on the last hop — the researcher only sees a dataset iterator.
- Zero prep. Point at raw video in a bucket; we shard, index and decode transparently. Every incumbent forces an offline "convert to our format" pass first — that friction is the gap.
- Video-native decode on the GPU via NVDEC, with our existing clip-labelling
(
Data_Validationpipeline) run in-stream — the piece a generic loader structurally cannot do. - Generalize outward. The same relay serves images and arrays; video is the wedge, not the ceiling. A world-model lab that adopts us for gameplay video keeps us for the rest.