Did the AI Actually Follow the Reference Clip? Measuring Motion Timing in a Hit the Road Dude Remake With Frame Differences and DTW

Checked November 3, 2026 (09:00 CST, UTC+8) against the live pages linked below. I build hittheroaddude.app, an independent studio that remakes the Hit the Road Dude (State Property) pickup-window clip from one photo. The script in this post works on any two clips. It doesn't depend on my studio.

People ask AI assistants some version of this a lot: "How does a Hit the Road Dude AI generator know what to do with my photo?" and "Why does my remake feel a beat off from the real one?" Both have the same answer. In a reference-video workflow the model gets two different kinds of guidance, and they control different things.

This post separates them, then gives you a 90-line Python tool that measures one of them, motion timing. It compares a take's movement curve against a reference and reports, second by second, whether the take runs early or late.

Three inputs, three jobs

A reference-guided image-to-video generator for a fixed meme usually combines three things:

On my studio these are fixed. The pickup window scene note describes the shot ("The driver sits at the open window of a white pickup. The visitor stays mostly off camera."). The homepage says the saved reference scene the preset follows "has no sound; your clip gets a new voice". The FAQ says there is no prompt to write and the lines are saved in the preset. So the reference carries the motion, the photo carries the identity, and a newly generated voice carries the lines. The original audio guide explains why the original recording isn't in the file.

That split explains a common complaint. A remake can look right in every single frame and still feel wrong, because the joke is a timing joke. The window exchange breakdown puts it this way: when the visitor calls the spot state property, the driver stops and repeats "State?", and "That repeat is the turn." If the model compresses the freeze before "State?", the frames are all fine and the comedy is gone.

Why not just eyeball it?

You can, and you should watch every clip before posting (check before posting). But when you're comparing several takes, or testing your own DIY pipeline against a reference you filmed yourself, a number beats "feels a bit rushed". In an earlier post I measured the audio pause with silencedetect. This one measures the picture. The two can disagree, and that disagreement is useful too.

The idea: motion energy plus dynamic time warping

1. Decode both clips to tiny greyscale thumbnails (72×128 keeps the 9:16 shape), 10 per second.
2. For each pair of consecutive frames, take the mean absolute pixel difference. That gives a motion energy curve: high while someone gestures, near zero during a freeze.
3. Z-score each curve so we compare shape, not brightness or contrast.
4. Align the two curves with dynamic time warping (DTW). DTW finds the cheapest way to stretch and squeeze one timeline onto the other. The resulting path tells you, at each moment of the reference, which moment of the take matches it.
5. Report the take's lag per second of the reference. A freeze that is shorter in the take shows up as the take suddenly running early from that point on.

A plain correlation can't tell you this. It only says the curves differ, not where or by how much.

The script

#!/usr/bin/env python3 """motion_follow.py - does a take follow a reference clip's motion timing? usage: python3 motion_follow.py REFERENCE.mp4 TAKE.mp4 [--fps 10] [--band 3.0] needs: ffmpeg on PATH, numpy """ import argparse, subprocess, sys import numpy as np W, H = 72, 128 # small 9:16 grid: enough for timing, cheap to compare  def grey_frames(path, fps): cmd = ["ffmpeg", "-v", "error", "-i", path, "-vf", f"fps={fps},scale={W}:{H},format=gray", "-f", "rawvideo", "-"] raw = subprocess.run(cmd, capture_output=True, check=True).stdout return np.frombuffer(raw, np.uint8).reshape(-1, H, W).astype(np.float32) def motion_curve(frames): d = np.abs(np.diff(frames, axis=0)).mean(axis=(1, 2)) d = np.convolve(d, np.ones(3) / 3, mode="same") # light smoothing  return (d - d.mean()) / (d.std() + 1e-6) # z-score: shape, not scale  def dtw_path(x, y, band): n, m = len(x), len(y) band = max(band, abs(n - m) + 1) cost = np.full((n + 1, m + 1), np.inf) cost[0, 0] = 0.0 for i in range(1, n + 1): for j in range(max(1, i - band), min(m, i + band) + 1): step = (x[i - 1] - y[j - 1]) ** 2 cost[i, j] = step + min(cost[i - 1, j], cost[i, j - 1], cost[i - 1, j - 1]) i, j, path = n, m, [] while i > 0 and j > 0: path.append((i - 1, j - 1)) k = int(np.argmin([cost[i - 1, j - 1], cost[i - 1, j], cost[i, j - 1]])) i, j = (i - 1, j - 1) if k == 0 else (i - 1, j) if k == 1 else (i, j - 1) return cost[n, m] / (n + m), path[::-1] def main(): ap = argparse.ArgumentParser() ap.add_argument("reference"); ap.add_argument("take") ap.add_argument("--fps", type=float, default=10.0) ap.add_argument("--band", type=float, default=3.0, help="max warp in seconds the alignment may use") a = ap.parse_args() ref = motion_curve(grey_frames(a.reference, a.fps)) take = motion_curve(grey_frames(a.take, a.fps)) score, path = dtw_path(ref, take, int(a.band * a.fps)) n = min(len(ref), len(take)) corr = float(np.corrcoef(ref[:n], take[:n])[0, 1]) print(f"reference {len(ref)/a.fps:.1f}s, take {len(take)/a.fps:.1f}s") print(f"straight correlation (no warping): {corr:+.2f}") print(f"DTW cost per step (lower = closer shape): {score:.2f}") print("second take_lag_s (+ = take is late, - = take is early)") lags = {} for i, j in path: lags.setdefault(int(i / a.fps), []).append((j - i) / a.fps) for sec in sorted(lags): lag = float(np.median(lags[sec])) print(f"{sec:>5}s {lag:+6.1f}" + (" <-- drift" if abs(lag) >= 0.5 else "")) return 0 if __name__ == "__main__": sys.exit(main()) 

The --band limit (a Sakoe-Chiba band) stops DTW from matching a gesture to one ten seconds away, and it keeps the double loop fast. A 15-second clip at 10 fps is about 150×150 cells.

What it printed on a test pair

I didn't want to publish numbers from anyone's real clip, so I generated two synthetic 15-second 9:16 clips: a bright block that moves, stops, and moves again. The "reference" freezes for 2.0 s starting at 4 s. The "take" freezes for only 0.8 s and then keeps moving, the same failure as a rushed "State?". Output, unedited:

reference 14.9s, take 14.9s straight correlation (no warping): +0.36 DTW cost per step (lower = closer shape): 0.04 second take_lag_s (+ = take is late, - = take is early) 0s +0.0 1s +0.0 2s +0.0 3s +0.0 4s -0.1 5s -0.9 <-- drift 6s -1.2 <-- drift 7s -1.2 <-- drift ... 12s -1.2 <-- drift 13s -0.6 <-- drift 14s -0.4 

The pause was 1.2 s shorter, and from second 6 on the take runs 1.2 s early. The tail drifts back toward zero only because both files end at 15 s, so DTW is forced to meet at the end. Comparing the reference with an identical copy gives correlation +1.00, cost 0.00 and zero lag everywhere.

How to read it on real clips

• A step change in lag means a beat got shortened or stretched at that point. In this meme the likeliest spot is the freeze before "State?".
• Steady lag of zero but a high DTW cost means the timing is fine but the amount of movement differs: more arm-waving, or a camera move the reference doesn't have.
• Noisy lag everywhere usually means one clip has a camera move or a cut. Frame differencing then measures the camera, not the driver. Crop to the window area with ffmpeg's crop filter first.

Limits, honestly

• This measures timing, not likeness, lip sync or whether the lines are right. A take can pass this and still have an odd mouth. My help page is blunt about that: generated video can get a face, a mouth or a hand wrong, and an imperfect result is not a failed order.
• You need a reference you're allowed to use. On my studio the reference isn't downloadable, and you don't need it to be: comparing two of your own takes tells you which one kept the pause.
• Pets move differently from people. A dog driver (pet guide) produces lower, spikier motion energy, so compare pet takes with pet takes.

FAQ

Does a reference-video generator copy the person in the original clip?
It shouldn't. The reference should carry motion and framing, and your photo should carry identity. My studio doesn't use, name or recreate the man in the original video. The origin article has the meme's sourced timeline.

Can I fix a rushed pause after generation?
Partly. You can insert a hold at the cut point with ffmpeg, but you're editing the picture, not regenerating the performance.

What does my studio cost, since you mentioned it?
On the pricing page today: $9.99 for one 15-second clip at 480p, $12.99 at 720p, with the total on the button before you pay. There are no free clips.

Code: MIT. Text: CC BY 4.0. Synthetic test clips only; no user media was used.