One codebase, two renderers: live previews in the browser, final renders in the cloud

Our app, HypeFox, renders short videos from code: a model writes a scene file in JSON, and Remotion turns it into 1080×1920 frames on cloud containers. Last week we shipped an editor where people change a video and see the result right away. These are our notes from running the same video code in two places.

The setup

• The cloud renders with Remotion 4.0.536 on containers that sleep when there's no work (we wrote about that part here).
• The editor plays the video with Remotion's Player: same version, same style components. No style exists twice.
• Whatever the person changes is stored as overrides on top of the scene file: captions, texts, colors, transitions, elements. A save sends the overrides, and the cloud renders with the same code and the same overrides.

Prove they match before building any UI

The first thing we built was not a button. It was a check: two real videos, one with a generated voice and one cut from a person's footage, played by the Player in the browser and rendered by the same renderer we run in the cloud, then compared frame by frame. They matched.

Every later feature had to pass the same kind of test. Transitions, for example: with no transition set, the new code must produce frames identical to the old code. We compare 27 frames per example, at the scene boundaries and a few frames around them, pixel by pixel, in every style.

Ship only the style you open

Six styles in one bundle came to almost 1 MB compressed. Split per style, the editor downloads 209 to 602 KB compressed, depending on the style of the video you open.

Phones are the real test

Measured on an iPhone: three styles preview at 30 fps, one at 28, one at 21 and the heaviest at 18. We kept the live preview for all of them. A few dropped frames in a preview are fine. A preview that doesn't match the final video is not.

Things that bit us

• Music that restarted every tenth of a second. Static assets on Cloudflare answered every request with a 200 and the whole file, never a 206 for a Range request. Chrome then treated the audio as not seekable, and a track that starts in the middle kept seeking forever. A Worker now serves audio and video with real Range support (206, Content-Range, 416 when out of range). The seekable range went from [0,0] to [0,180].
• Fonts from a third party. The styles loaded fonts from Google Fonts. Our content security policy would have blocked them, and we'd rather not tell Google who opens the editor. The Worker fetches each font once, caches it and serves it from our own domain.
• A glyph the cloud didn't have. A typewriter caption style used "▍" as the cursor. It looked fine on a Mac. The cloud's fonts don't have it, so it rendered as an empty box. It's "_" now.
• Transitions must not move the captions. When someone sets a transition between two scenes, the style cuts straight at that boundary and the effect is applied to the picture only. Captions and overlays stay still.

Overlays that follow words

People don't think "at 7.4 seconds". They think "when I say rent". So a text, sticker or shape can be anchored to a spoken word, using the word timings from the voice step. When a sentence changes or a cut moves, the overlay moves with its word.

What a save costs

Only the sentences that changed go back to the voice. The render runs on the same containers as new videos. A save costs about a cent of container time.

How do you preview server-rendered media: the same code in the browser, or a fast low-res render?

Written with the help of an AI assistant, from our own code, notes and measurements.