Multimodal’s next race is measured in milliseconds.

Send 1 minute of video to a frontier model and it bills you roughly 18,000 tokens before it says a word. That is not physics. It is a design choice: today’s vision-language models decode every frame back to pixels, slice each one into hundreds of tokens, and make attention rediscover motion the video file had already written down. Per Google’s own docs, the Gemini API samples video at 1 frame per second and charges about 300 tokens for each of those seconds, 258 for the frame and 32 for the audio.

In the last 4 days of July, 3 releases attacked that waste from 3 different directions. On July 27, Mage-VL proposed reading the compressed video stream itself instead of decoding it. On July 29, TurboVLA showed a robot policy acting 32 times per second on 0.2B parameters and a single consumer GPU. On July 30, Google DeepMind shipped Gemini Robotics 2, including a variant that runs on the robot with no cloud round trip at all. Different labs, different problems, one shared bet: the next frontier metric is not a benchmark score. It is the clock.

The codec already did the work

Here is the part most people building on video APIs never think about. A video file is not a stack of pictures. Codecs like H.264 store a full image only occasionally (an I-frame), then describe every following frame as changes: motion vectors that say “this block of pixels moved 3 pixels left” and residuals that patch whatever the motion guess got wrong. Compression works because most of every frame is the previous frame. Your camera computes this on the fly. Every video you have ever uploaded already contains a precise, machine-readable map of what moved, where, and how much.

The standard vision-language pipeline throws that map away. It decodes the stream back to raw RGB frames, tokenizes each frame from scratch, and then spends attention flops re-deriving the motion the codec had already computed. Mage-VL (arXiv 2607.24904, submitted July 27) skips the decode. Its tokenizer, Mage-ViT, consumes I-frames plus the motion vectors and residual energy of the frames between them, and spends tokens only where the residual energy says something actually changed. The result, as reported by the authors: over 75% fewer visual tokens at the same 16x16 patch granularity, up to 3.5x wall-clock speedup, performance matching Qwen3-VL-4B on static benchmarks, and gains over a 15B reasoning-vision baseline on video understanding.

How to cite: The Multimodal Society, “Tokens per Second: Multimodal AI’s New Race Is Real Time,” August 2026. multimodalsociety.com/blog/tokens-per-second

The second idea in the paper matters as much as the tokenizer. Mage-VL splits the model into a lightweight event gate that watches the stream continuously and a heavier causal decoder that only engages when the gate fires. The authors frame it as a fast System 1 and a deliberate System 2. In product terms: the model stops answering on a schedule and starts answering on events. That is the difference between a chatbot you show clips to and a system that can watch.

Small enough to act on time

Perception is half the loop. The other half is acting on what you saw before it stops being true. The conventional vision-language-action recipe routes camera frames into a large language model and asks it to emit motor commands, which puts billions of parameters between the eye and the hand on every single step. TurboVLA (arXiv 2607.27205, July 29) deletes the detour. Vision and language are encoded separately, exchange information through a lightweight bidirectional layer, and feed a compact action decoder directly. The whole model is 0.2B parameters. It runs at 32 Hz on an RTX 4090 in under 1 GB of VRAM, about 31 milliseconds per decision, and still reports 97.7% average success on the LIBERO manipulation benchmark, with code on GitHub.

One day later, Gemini Robotics 2 arrived as 3 coordinated models. The flagship is a vision-language-action model that controls a full humanoid, walking, balancing, crouching, and manipulating with 5-fingered hands, and also drives conventional dual-arm rigs. Above it sits Gemini Robotics ER 2, the embodied reasoning layer: it watches execution in real time, tracks task progress, finds specific moments in video with a reported 91.3% accuracy at 0.96 seconds of latency, and orchestrates the policies below it along with APIs and search. Below both sits Gemini Robotics On-Device 2, which runs without a cloud connection and adapts to a new dual-arm robot with fewer than 200 demonstrations. Google’s claim of 4x execution speed over competing stacks is self-reported and should be read that way. The architecture is the story: a slow deliberate brain in the cloud, a fast reflex policy at the edge, and an explicit latency budget between them.

Why the clock is the new benchmark

For 3 years the multimodal race was scored on quality: sharper frames, longer clips, higher win rates. Last week we argued that generation is consolidating into everything-models. This week’s cluster shows the understanding side consolidating around a different scarcity. Once every lab can describe a video accurately, the differentiator becomes who can do it while the video is still happening, on hardware the customer already owns, at a price that survives being always on.

Run the numbers on always-on. At the Gemini API’s default rates, watching a single camera feed costs about 18,000 tokens per minute, over a million tokens an hour, before the model produces a single useful sentence. Google already ships a pressure valve: a low-resolution mode that cuts frames from 258 tokens to 66, about 100 tokens per second with audio. Mage-VL’s result says another 4x or more is sitting in the codec, because most seconds of most videos contain almost no new information, and the bitstream tells you exactly which seconds those are. Continuous perception is currently priced like continuous novelty. It is not, and the first API to price the difference changes what can be built on it.

The event-gate pattern is the piece you can steal today, without waiting for anyone’s weights. A small cheap model watches the stream at high frequency and decides when something matters; a large expensive model is woken only for those moments. Mage-VL builds the gate into the architecture. Gemini Robotics 2 builds it into the product, with a reflex policy at the edge and a reasoning model above it. Nothing stops an application developer from building it out of 2 off-the-shelf models this afternoon: the pattern is a latency-and-cost budget, and budgets do not need frontier access.

If you build

• Budget video in tokens per second, not tokens per request. 300 per second at default, about 100 at low resolution. Most video tasks survive the low-resolution setting, and it is a 3x cost cut that takes 1 parameter. Measure your task at low resolution before assuming you need the detail.
• Build the event gate now. A small vision model sampling the stream, escalating to a frontier model only on change, is the Mage-VL architecture approximated with rented parts. For monitoring, logging, and assistant products, this is the difference between viable and absurd unit economics.
• Stop routing reflexes through a giant model. TurboVLA’s result, 97.7% on LIBERO from 0.2B parameters, says the capability floor for narrow, fast, well-scoped visuomotor tasks is far lower than the frontier-model default assumes. If your loop needs to close in 30 milliseconds, design for a small specialist and let a big model supervise at 1 Hz, which is exactly the split Gemini Robotics 2 ships.
• Watch for codec-native tokenization in production APIs. If it lands, always-on video input gets several times cheaper without a model upgrade, and product categories that look unaffordable today, ambient perception first among them, reprice overnight.

If you create

• Tools are about to start watching you work. An assistant that sees your timeline, your viewfinder, or your canvas continuously only becomes shippable when watching is nearly free. That is precisely the number this week’s research is driving down. Expect the first credible on-set and in-edit assistants to be event-gated systems that speak rarely and on cue.
• Moment-finding is a logging assistant. ER 2’s headline skill, locating a described moment in video at 91.3% accuracy in about a second, is robotics infrastructure that happens to describe dailies work. “Find every take where she looks camera left before the line” is the same query. Footage search stops being a manual craft before generation replaces any of it.
• Real-time worlds are becoming an exhibition medium. Genie 3 at 24 fps and RELIC at 16 fps are generated worlds running at frame rates an audience can stand inside. Installations, live visuals, and performance work that responds to a room are moving from render-farm territory to a single GPU. The craft constraint to learn is the latency budget, the same one game designers have carried for decades.
• Direction transfers to live systems. The session-directing skill we flagged for conversational video compounds here: directing a system that responds in under a second is closer to directing a performer than operating a renderer. The creative professionals who practice giving notes in real time will be first in line when the tools start listening at 30 Hz.

A prediction you can check us on

Claims without dates are marketing, so here is ours. By August 2027, at least one frontier API will offer an always-on video mode priced at or below the equivalent of 100 tokens per second of watched footage, and its technical notes will describe some form of change-triggered gating rather than uniform per-frame billing. If instead flat per-frame pricing is still the only option a year out, then continuous perception stayed a robotics luxury and the codec-native bet stalled, and we will say so in this space. Either way, watch the spec sheets: when a lab starts advertising a rate instead of a resolution, you are looking at the new race.