Paper guides
How to actually read these papers
Each guide starts the way a human would solve the problem — a concrete case study — then maps every step onto the paper’s modules. Diagrams, comparison charts, and embedded lectures sit next to the mapping. Jargon comes second.
SmolVLA + LeRobot
Case study: cube-in-bowl on a cheap arm — VLM tokens, flow-matched action chunk, async client so the arm does not stall.
VGGT: Visual Geometry Grounded Transformer
Case study: cameras, depth, and tracks from phone photos in one transformer pass — not match / triangulate / bundle-adjust.
ACT: Action Chunking with Transformers
Case study: opening a condiment cup — predict a chunk of $k$ joints, one CVAE mode, temporal ensemble — not one command per 20 ms.
STLight
Case study: past $T$ frames → future $T'$ without unrolling a ConvLSTM — joint space–time convolution and a pixel-shuffle head.
JEPA · LeCun 2022
Case study: reach for a mug. A generative model would reconstruct the next RGB frame $y$. A JEPA trains $P(E(x),z)\approx E(y)$ and plans in that latent.
I-JEPA
Case study: a notebook covering a dog’s head — predict $E(y)$ on the hidden blocks, not MAE pixels. Context ViT, EMA targets, multi-block masking, no SimCLR jitter.
V-JEPA
Case study: a sleeve occludes a pour — mask a space–time tube, predict $E(y)$, freeze the backbone for Kinetics and Something-Something.
V-JEPA 2
Case study: 1M-hour action-free JEPA, then 62 h Droid to learn $P(s_t,a_t)\approx s_{t+1}$, then zero-shot Franka MPC toward an image goal.
Constellation
Foundations these papers stand on
Not full guides (yet). Each is a spoke on one of the knowledge graphs above. Open the arXiv if the node is load-bearing for what you are trying to build.
Flow Matching · Lipman et al.
Velocity fields instead of scores. Why SmolVLA’s action expert can be 0.75d and still sample inside a control cycle.
DUSt3R
Two views → point maps, then global alignment. Read this to see the post-process VGGT deleted.
DINOv2
The patch encoder VGGT barely modifies. Alternating attention sits on top of this.
Diffusion Policy · Chi et al.
Same chunking instinct, diffusion instead of CVAE. LeRobot ships both next to ACT.
PredRNN
Zigzag spatiotemporal LSTM. The unroll whose FLOPs STLight claims 2% of on KTH.
OpenVLA
7B discrete action tokens. The heavy open VLA SmolVLA is sized against.