Video as code: Remotion, and where an LLM actually helps

Remotion lets you describe a video as React components and render it frame by frame to a file. You write a composition, every component gets told which frame it is on, and a headless browser draws each one in turn.

It is a genuinely different way to think about video, and the difference that matters is not that it is programmable. It is that video becomes text in a repository: diffable, reviewable, and — this is the part that changes what is possible — generatable.

Why code beats a timeline for anything repeated

If you are making one video, an editor is faster. Nothing about writing React competes with dragging a clip.

The calculus inverts the moment you are making the same shape of video repeatedly. A composition is a template with real logic in it. Change the brand colour once, and every video ever made from that template changes. Add a caption style, and it applies retroactively. Fix a spacing bug, re-render, done.

Two properties follow that are hard to get any other way:

It is reviewable. A change to a video is a diff. You can see that someone altered the title timing because it says so in the patch, and you can revert it the same way you revert anything else.

It is deterministic. The same inputs produce the same frames. Which means a rebuilt machine produces an identical video, and a regression is a real regression rather than a vague sense that it used to look better.

The division of labour that works

The obvious idea is to ask a language model to write the composition. I tried this. It is the wrong shape.

Models will happily emit React that renders. What they will not reliably emit is motion that feels right — the timing, easing and overlap that separate something watchable from something that looks like a slide deck with transitions. And every mistake in that category is invisible in the code and obvious in the output, so you only find it by rendering, which is slow.

What works far better:

The model writes the words. The code owns the timing.

The narration script, the caption text, the on-screen copy — these are language problems, and language models are extremely good at them. Get the script first. Then generate the narration audio, measure it, and derive every timing in the composition from the real measured duration.

That ordering matters more than anything else here. If you write the animation first and generate the voiceover second, you are forever nudging keyframes to fit audio that does not quite match. If the audio comes first and the composition reads its actual length, the two can never drift apart. Scenes become “as long as this sentence takes”, which is both correct and self-maintaining.

Structured output is what makes it reliable

The other place a model earns its keep is deciding the shape of a video: which scenes, in what order, carrying what content.

The thing that turns this from a novelty into something dependable is asking for structured output — a schema the response must conform to — rather than prose you then parse. You get an object with the fields you asked for, validated, or you get an error. No regex over prose, no “sometimes it wraps the JSON in a code fence”.

Two rules I would not now break:

Only offer choices that are actually available. I had a bug where the model picked a scene type the renderer had no data to fill. It rendered — as a heading over an empty screen, for five seconds, while the narration described content that was not there. It looked like the video was broken, because it was.

The fix was not “populate the data”. It was to compute the list of scene types that have data first, interpolate that list into the prompt, and then filter the response against it anyway. Both layers matter: the prompt keeps the words honest, the filter keeps the picture honest.

Enforce limits in code, not in the prompt. I asked for scripts under a certain length and got, on average, scripts under that length — with a long tail that overran badly. The prompt was advice. Nothing checked it, and the composition took its duration from the measured audio, so an overlong script silently produced an overlong video.

Any constraint that actually matters needs a deterministic check after generation. If it is only in the prompt, it is a preference.

Budget for the gaps you forgot

A small, concrete thing that cost me an afternoon. Each scene had a short pause after its narration — a fraction of a second so lines do not run together, plus a slightly longer one at the end.

Individually trivial. Across seven scenes it added up to several seconds, which meant a script sized to exactly fill the target duration produced a video that overran it. The budget was being computed against the total length rather than the length minus the padding the renderer would add.

The general form: when you derive a budget from a duration, subtract the overhead your own code introduces. It is obvious in hindsight and invisible until you measure the output rather than trusting the plan.

Render times and the feedback loop

Remotion renders in a headless browser, and it parallelises across cores well. On a machine with enough of them, a short vertical video is a couple of minutes.

That is fast enough to be practical and slow enough that you do not want it in a tight loop. Two things helped:

Extract stills, not videos, when checking layout. Rendering a handful of individual frames takes seconds and answers most “does this look right” questions. Keep full renders for checking motion.

Typecheck before rendering, always. The composition is TypeScript. A type error caught in a second is a type error you did not find four minutes into a render.

Where it does not fit

Remotion is not an editor and it is bad at being one. Anything driven by footage — cutting to a performance, reacting to what is on screen, matching a feel — wants a timeline and a human. Trying to express that as code is a lot of work to arrive somewhere worse.

The sweet spot is designed video: motion graphics, explainers, data-driven pieces, anything templated. Things where the structure is known in advance and the content varies. For those it is not merely a reasonable option, it is a better one, and the fact that a language model can fill in the content makes the whole thing move at a speed that has genuinely surprised me.