Research collabs RC01 · 16 July 2026

Michele Zhu: from semantic video to a trajectory codec

Prospective collaboration brief: whether semantic communication can become a layered video + action format that preserves better training evidence per transmitted byte for action-conditioned world models.

Relationship status — interview hypothesis, not an active collaboration. This note records the technical fit to test in conversation and the experiment that could turn the hypothesis into evidence.

1 · The short verdict

The direction is coherent; the demonstrated systems are adjacent, not equivalent. Zhu has worked on compact learned video state, rate–distortion trade-offs, shared semantic knowledge, and error-aware transmission. Those are useful foundations for an experience codec. Neither work, however, transports causal actions or tests whether compressed evidence trains a better world model.

Strong fitsemantic compression · compact motion state · learned encoder/decoder · adaptive transmission

Useful fitquantization · source coding · temporal error correction · shared model assumptions

Not yet shownaction-conditioned dynamics · open-world scenes · rare-event preservation · downstream model training

Separate systems questionstable bitstream syntax · random access · model versioning · hardware and latency

2 · What the two works actually demonstrate

Work	Demonstrated mechanism	Why it connects	Boundary
Semantic Communication for Video Conferencing (2023 thesis)	A shared source image plus per-frame head rotation, translation, and expression deformation over 15 learned 3D keypoints; quantization, Huffman coding, and temporal semantic correction.	It sends an anchor state followed by compact state changes instead of repeatedly coding pixels.	256×256 VoxCeleb talking heads; motion is inferred from video; the source image, neural decoder, and codebook are shared assumptions.
Semantic Communications via Features Identification (ICC 2025)	A teacher sends feature position/value packets until an apprentice identifies a semantic class above a confidence threshold.	It introduces progressive, receiver-aware transmission and the useful split between learnable structure and memorizable residual information.	The experiment is Imagenette classification from ResNet-50 features: no video reconstruction, temporal dynamics, actions, or implemented learnable/memorizable split.

The thesis reports a selected codebook at 0.00345 bits per pixel—about 226 bits per 256×256 frame—and roughly 7× fewer bits than its H.264 CRF 36 comparison at similar LPIPS. The 2025 paper reports about 80% class identification with 18% of the feature bits, or about 90% with 45–50%. These are promising in-domain rate–distortion results, not end-to-end world-model data results.

Attribution boundary — the thesis adopts the learned talking-head and 3D-keypoint architecture from Wang, Mallya & Liu's CVPR 2021 system. Its clearest added work is the quantized Huffman source code and semantic error-correction analysis.

3 · The bridge to our data problem

Anchor evidencekey observation, calibration, scene context

→

Semantic statecompact visual or geometric tokens

Exact actionstimestamped controls in a declared action space

Novel residualevidence the current representation cannot explain

→

Trainer / decoderconsume tokens directly or reconstruct an auditable view

The strongest shared idea is not “replace video with semantics.” It is the hybrid split:

Predictable structure can be represented compactly and transmitted progressively.
Actions are cheap but causally essential, so they remain exact and clock-aligned.
Unpredictable residuals receive more bytes because novelty is valuable training evidence.

The 2025 paper calls the first and third streams learnable and memorizable, but also identifies their reliable separation as an open problem. For us, that boundary is the central research problem: the codec must know what it is unsafe to discard.

4 · Where the mapping breaks

Current objective	World-model requirement	Necessary change
Perceptually plausible reconstruction	Evidence-faithful state transitions	Measure predictive and control sufficiency, not LPIPS alone.
Motion estimated from the resulting frame	The intervention that caused the next state	Record `(o_t, a_t, o_t+1)` on one clock; do not substitute optical motion for `a_t`.
Closed-domain faces or ten image classes	Open-world objects, physics, occlusion, UI, and rare failures	Use a layered residual or raw escape path when semantic confidence is low.
A shared, fixed decoder and semantic base	Datasets that outlive today's model	Version the schema, encoder, decoder, calibration, and provenance; retain migration evidence.
Errors may be smoothed into plausible motion	Training data must not silently invent events	Expose uncertainty and corruption masks; never treat generated fill as measured evidence.

Non-negotiable distinction — motion is not action. Head pose or optical flow says what changed in the observation. A controller state, mouse delta, robot command, or game API event says what intervention preceded the change. Only the latter supports causal counterfactuals.

5 · A candidate trajectory chunk

trajectory_chunk {
  schema_version, semantic_model_id
  t0, clock_domain, sensor_calibration
  anchor_observation
  actions[]          // timestamp, action-space id, exact value
  semantic_tokens[]  // compact state / motion representation
  residuals[]        // novel or low-confidence measured evidence
  uncertainty_mask, corruption_mask
  provenance, checksums
}

This is a compound, versioned experience format before it is a new codec. It earns the word codec only when the bitstream syntax, decoder behavior, random access, error containment, and compatibility contract are stable. Initially, the pixel evidence layer can use AV1 while the research focuses on semantic tokens, novelty allocation, and exact action synchronization.

6 · The experiment that decides the collaboration

Collect the same action-labelled trajectories once, then package them three ways under the same total byte budget:

A · Conventional baseline: AV1 video plus a lossless action sidecar.
B · Semantic-only: learned state tokens plus actions, with no residual escape.
C · Layered proposal: anchor/key observations, learned state, exact actions, and novelty-triggered residuals.

Train the same small action-conditioned predictor on each package. Compare next-state prediction, long-rollout consistency, downstream control, rare-event recall, out-of-distribution transfer, and performance after a semantic-model upgrade. C wins only if it improves useful world-model performance per stored or transmitted byte without B's expected loss of rare evidence.

7 · Interview agenda

How would you redesign your representation for (video, action) trajectories rather than inferred facial motion?
What distortion function replaces LPIPS when the consumer is a future world model?
How do you prevent the generative decoder from deleting rare events or adding false evidence?
Would the world model consume transmitted tokens directly or reconstructed pixels? What is the migration story?
How do sender and receiver negotiate semantic-base and decoder versions?
Did the reported rates include source images, model/codebook distribution, headers, and random-access cost?
What six-week prototype would falsify the layered-codec hypothesis?

Positive signal — a strong answer should reject perceptual quality as the sole objective, preserve actions exactly, introduce a measured residual escape path, and propose an equal-byte downstream evaluation. Treating a plausible GAN reconstruction as ground truth would be the central warning sign.

Primary sources: Zhu's 2023 master's thesis · Mariani, Zhu & Magarini, ICC 2025 · Wang, Mallya & Liu, CVPR 2021.