Methods to Measure Video Similarity: 6 Strategies Examined

Two brief clips. One query: how alike do they appear? Sounds trivial, it isn’t, and I discovered that the gradual method. 

My setup: one reference clip, eight others to rank in opposition to it, all waterfalls (extra on why in a second). I figured this was a day job, seize a mannequin, compute a quantity, transfer on. As a substitute I watched supposedly-smart strategies rank near-identical clips in nonsense orders, and the one which regarded greatest on paper was too gradual to truly use. 

So I benchmarked six strategies, similar clips, similar guidelines, and judged them accuracy first. A fallacious reply in a millisecond remains to be fallacious, so velocity solely will get a say as soon as a way proves it might probably rank appropriately. Accuracy first, velocity because the tiebreaker. Right here’s what I discovered. 

Why that is tougher than it sounds

My first intuition was to only examine pixels. That’s the lure: you’re truly evaluating which means, the topic, colours, mild, whether or not the water crashes or trickles. 

Chase which means and also you’re choosing from three households, every costing you one thing. Embeddings pattern frames by means of a mannequin that is aware of what pictures imply, good, however prices milliseconds or API credit. Fingerprints crush every body to a tiny code, nearly free and prompt, nearly blind to something delicate. Full multimodal LLMs take the entire video and hand you an opinion, they see essentially the most, in addition they break essentially the most. 

Pace, accuracy, price. You get two. That’s the entire purpose I benchmarked as an alternative of arguing about it. 

The six contenders

Three of those run free on my laptop computer. Three invoice me per name. That cut up mattered extra to me than the accuracy numbers. 

Establishing a good combat

Most benchmarks cheat by testing on a simple set, so I made mine imply on function. The reference is a tropical waterfall, sunbeams and moist greenery, and all eight check clips are waterfalls too. That forces each technique to win on the positive element, coloration solid, mild course, framing, movement, as an alternative of simply recognizing “there’s water on this one.” 

Similar enter for everybody: six frames per clip, evenly spaced, shrunk to 384×216. Sampling just a few frames as an alternative of all of them is regular and barely prices you high quality. 

The annoying half: no human labels, no funds to make any. So I faked a good consensus, averaged the primary 5 strategies’ scores per clip. Two clips got here out on high, together with one I’ll name Pattern 4 (proven within the 2nd video under) and the unique video (the first video); all the things else will get graded in opposition to. Imperfect, however defensible. 

The strategies, and the place every one broke

I’m skipping the neat pros-and-cons bins. They didn’t break neatly. 

GPT Imaginative and prescient

GPT Imaginative and prescient was the one I truly preferred studying, hand it just a few frames and it judges theme, coloration, and temper, writing again one thing like “differed considerably in visible theme, coloration palette, and total temper.” However the numbers beneath had been mush, all the things scored 50 to 80, bunched collectively and wobbling between runs. Nice at explaining itself, dangerous at rating eight near-twins. 

Right here’s the precise name, trimmed down: 

# Seize just a few frames from every video, present them facet by facet to
# GPT-4o-mini, and simply ask it to attain how comparable they appear.

def score_with_gpt_vision(ref_frames, test_frames, api_key):
    shopper = OpenAI(api_key=api_key)

    ref_imgs = frames_to_jpeg_b64(ref_frames[:3])
    test_imgs = frames_to_jpeg_b64(test_frames[:3])

    response = shopper.chat.completions.create(
        mannequin="gpt-4o-mini",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "REFERENCE VIDEO:"},
                    *[image_message(img) for img in ref_imgs],
                    {"kind": "textual content", "textual content": "TEST VIDEO:"},
                    *[image_message(img) for img in test_imgs],
                    {
                        "kind": "textual content",
                        "textual content": "Rating how intently these match, 0 to 100, JSON solely.",
                    },
                ],
            }
        ],
    )

    knowledge = json.hundreds(response.decisions[0].message.content material)

    return knowledge["score"], knowledge["feedback"]

Gemini Flash

Gemini Flash on the complete video is the one one which watches actual movement, not stills, pacing and digital camera drift included, a giant edge on paper. Then actuality confirmed up: slowest by a mile, and mid-test one clip threw a 503, the fallback hit a 429, and that clip by no means received a rating in any respect. Dealbreaker for something dwell. 

Right here’s what makes this one completely different, code-wise: 

# Add each full movies and simply ask Gemini to observe and examine.
# The one approach right here that truly sees movement, not simply stills.

def score_with_gemini_flash(ref_path, test_path, api_key):
    shopper = genai.Shopper(api_key=api_key)

    ref_video = shopper.information.add(file=ref_path)
    test_video = shopper.information.add(file=test_path)

    # Watch for each information to depart PROCESSING state earlier than utilizing them

    immediate = "Examine these two movies for visible similarity. Rating 0-100, JSON solely."

    response = shopper.fashions.generate_content(
        mannequin="gemini-2.5-flash",
        contents=[prompt, ref_video, test_video],
        config={"response_mime_type": "software/json"},
    )

    knowledge = json.hundreds(response.textual content)

    return knowledge["score"], knowledge["feedback"]

CLIP

CLIP was my guess getting in: every body turns into a vector, you common them, take the cosine. Runs domestically in below a second and gave the steadiest scores within the check, although all the things clusters within the excessive 80s and low 90s even for clips that aren’t shut. Nice for rating, ineffective in order for you “you scored 88%” to imply something by itself. 

Right here’s the entire approach in code: 

# Flip each body right into a CLIP vector, common them into one vector
# per video, then simply take the cosine between the 2.

def embed_video_with_clip(frames, mannequin, preprocess):
    pictures = [preprocess(to_pil_image(f)) for f in frames]

    with torch.no_grad():
        vectors = mannequin.encode_image(torch.stack(pictures))

    vectors = vectors / vectors.norm(dim=-1, keepdim=True)

    # One vector for the entire video
    return vectors.imply(dim=0).numpy()


ref_vec = embed_video_with_clip(ref_frames, mannequin, preprocess)
test_vec = embed_video_with_clip(test_frames, mannequin, preprocess)

similarity = cosine_similarity(ref_vec, test_vec)

No API name in sight. As soon as the mannequin’s downloaded, this all runs by yourself machine. 

Perceptual hash is fifty occasions quicker than anything, no mannequin to even load. As a choose although, almost ineffective right here, it’s a bouncer, not a critic. (Precise quantity developing, funnier than I anticipated.) 

And right here’s all the factor, no mannequin required: 

# Hash each body right down to a tiny fingerprint, then measure
# what number of bits differ. No mannequin, no API, simply bit-counting.

def phash_similarity(ref_frames, test_frames):
    ref_hashes = [imagehash.phash(to_pil_image(f)) for f in ref_frames]
    test_hashes = [imagehash.phash(to_pil_image(f)) for f in test_frames]

    similarities = []

    for rh in ref_hashes:
        # Smallest Hamming distance
        closest = min(rh - th for th in test_hashes)

        # 64-bit hash
        similarities.append(1.0 - closest / 64)

    return sum(similarities) / len(similarities)

That’s the entire bouncer. Quick as a result of it isn’t truly something, simply counting flipped bits.  

CV multi-metric is what I’d construct to see inside a rating, 4 old-school indicators weighted by hand: 

composite = 0.30 * coloration  # HSV histogram 
       + 0.35 * struct # SSIM 
       + 0.20 * temporal   # temporal coloration profile 
       + 0.15 * edge   # edge density

One quirk that confirmed up: completely different metrics can wildly disagree on the identical clip. SSIM punishes any framing shift onerous, whereas a temporal color-profile examine barely notices it, so the identical video can rating a 99 on one sign and an 18 on one other. 

Gemini Embedding 2 is CLIP’s thought after it grew up. Similar transfer, embed then pool then cosine, however the embedding comes from a mannequin constructed to deal with picture, video and textual content in a single 3072-dimensional house. 

Right here’s what that truly seems to be like in code, trimmed right down to the half that issues: 

# Simply learn the entire video file and hand it to Gemini as one blob.
# No frames, no pooling, one API name per video.

def embed_full_video(video_path, api_key):
    with open(video_path, "rb") as f:
        video_b64 = base64.b64encode(f.learn()).decode()

    physique = {
        "content material": {
            "elements": [
                {
                    "inline_data": {
                        "mime_type": "video/mp4",
                        "data": video_b64,
                    }
                }
            ]
        }
    }

    resp = requests.submit(f"{EMBED_URL}?key={api_key}", json=physique)

    return np.array(resp.json()["embedding"]["values"])


# Embed each movies, then simply examine the 2 vectors

ref_vec = embed_full_video("Authentic.mp4", api_key)
test_vec = embed_full_video("sample_1.mp4", api_key)

cos = cosine_similarity(ref_vec, test_vec)

# Stretched onto a pleasant 0-100
rating = cosine_to_score(cos)

Does sampling frames even assist?

Fast facet check: if sampling frames works for CLIP, does it work the identical method for Gemini’s embedding mannequin? I examined one video at 4, 8, 16, 32, and 64 frames in opposition to sending the entire video in a single shot. 

4 to 32 frames buys 5 additional rating factors (65 to 70) however doubles the time (5.26s to 10.65s). Push to 64 frames and solely the time strikes, as much as 14.6s. Full video beats all of them on accuracy (73) whereas being quicker than something previous 4 frames. 

Right here’s the precise perform, stored so simple as I may make it: 

# Seize a handful of frames from the video, embed every one,
# then common them right into a single vector. Similar thought as CLIP,
# simply utilizing Gemini's embedding mannequin as an alternative.

def embed_video_by_frames(video_path, frame_count, api_key):
    frames = grab_frames(video_path, frame_count)
    vectors = []

    for body in frames:
        jpeg = frame_to_jpeg_b64(body)
        vec = embed_one_frame(jpeg, api_key)
        vectors.append(vec)

    # Common all of the body vectors into one video vector
    return np.imply(vectors, axis=0)

In order that settles it: extra frames doesn’t purchase higher accuracy previous a degree, only a longer wait. Full video wins each methods. 

Prices cash, prices just a few seconds. I walked in rolling my eyes on the latency. I walked out having shipped it. The subsequent part is why, and it’s all about accuracy. 

Accuracy first, as a result of a quick fallacious reply is ineffective

That is the one that truly issues, so I checked out it earlier than I regarded on the clock. 

Two questions. How most of the three consensus favorites did every technique discover? And the way intently did its full rating monitor the consensus total? 

There’s the perceptual hash quantity I promised: -8%. Damaging. Its rating leans very barely backwards from the consensus. A coin would have accomplished about as effectively. That alone knocks it out as a choose, regardless of how briskly it’s. 

After which the one which made me sit up. Gemini Embedding 2 discovered solely two of the three favorites, but it posted one of the best accuracy of the lot, 93%. 

How does the best-correlating technique miss a top-3 decide? As a result of the miss was a faux tie. 

Have a look at the uncooked numbers: the highest few clips all landed inside a hair of one another, whereas the following clip down had an actual, clear hole under them. 

So the mannequin ranked two very shut clips in a barely completely different order than the consensus did, and that one swap is what the top-3 depend punished it for. The rank correlation noticed straight by means of that. 

I actually assumed CLIP would high this desk. It didn’t. Not shut. 

So right here’s the place I stood after this desk. Three of the native strategies can rank appropriately, and two of the API strategies rank even higher. Perceptual hash is out. Solely now does velocity get a say, and solely among the many ones nonetheless standing. 

Then velocity, the tiebreaker

Now that I knew which strategies may truly rank the clips, velocity received to resolve between them. Not earlier than. A technique that may’t rank proper doesn’t earn speed-credit, it simply will get minimize. 

Quickest to slowest is roughly a 1,400x hole. Not a typo. Fourteen hundred occasions. 

Stare at that and the full-video LLM writes its personal rejection letter for something with an individual ready. It ranked effectively (88%), however intelligent doesn’t assist when intelligent takes 21 seconds and generally returns nothing in any respect. 

So it got here down to 2. CLIP is principally free in below a second. Gemini Embedding 2 is the extra correct one however prices about seven seconds. That’s the actual combat, and it’s the following part. 

The stunning winner, and the catch

I shipped Gemini Embedding 2. 

The case is 93%. Sit with what it means: the mannequin agreed with a panel of 5 impartial strategies extra tightly than these strategies agreed with their very own panel. One name, roughly the entire committee’s verdict. 

And the latency I griped about? I buried it. In case your interface already has the consumer busy for just a few seconds with one thing else, the scoring runs beneath and no one ever watches a spinner. 

Now the catch, as a result of that is the place most write-ups go quiet and faux there isn’t one. In case you can’t conceal the wait, the entire thing flips. 

Image a search field with somebody tapping their foot. Seven seconds there’s a catastrophe, and I’d ship native CLIP in a heartbeat and by no means really feel dangerous about it. 

Embedding 2 gained my constraints. Yours get a vote. Don’t let a weblog (this one included) make that decision for you. 

The calibration lure no one warns you about

Skip all the things else in order for you. Don’t skip this. It’s the half that quietly ate a day of mine. 

A cosine of 0.88 means nothing significantly insightful. So that you stretch it onto a 0 to 100 scale. High quality. 

The lure: each mannequin’s stretch is completely different. Copy one mannequin’s settings onto one other and your scores go haywire. 

Completely different fashions park “completely unrelated” at completely different cosines. CLIP places two unrelated pictures someplace round 0.2 to 0.5, so its ground sits at 0.20. Gemini Embedding 2 crams all the things larger and tighter. Even genuinely unrelated clips hardly ever dropped under 0.75, so its ground is 0.75. 

Reuse CLIP’s 0.2 ground on Gemini and watch each clip rating within the 90s. Then watch somebody file a bug that claims “the scores are damaged, everybody’s getting 90%.” 

The scores aren’t damaged. The ground is. That grievance is sort of by no means the mannequin, it’s nearly all the time this. Verify your individual mannequin’s actual cosine vary first, then set the ground. Borrowed numbers lie. 

So which one do you have to truly use?

There’s no “greatest.” Solely greatest to your case. Right here’s the cheat sheet I’d hand a teammate: 

The transfer that beats choosing one: stack them. Low cost filter up entrance (hash or CLIP), costly choose solely on the survivors. You get velocity and high quality each, and the invoice drops onerous. I didn’t construct it that method the primary time. I’d now. 

What’s modified by 2026

One trustworthy caveat: I leaned on CLIP because the native baseline of all articles as a result of the tooling’s all over the place, however it’s not the sharpest possibility anymore. DINOv3, educated on pictures alone with no captions, tends to beat it on fine-grained similarity now. SigLIP 2 and Meta’s Notion Encoder push retrieval additional nonetheless. 

And if movement issues to you, video-native encoders like V-JEPA 2 and VideoPrism now bake temporal construction proper into the embedding, the one factor frame-by-frame CLIP can by no means do. 

Actual takeaway: don’t weld your pipeline to 1 mannequin. Wrap it so you possibly can swap encoders in a day, as a result of a greater one lands each quarter now. Embedding 2 gained this spherical. The form (embed, pool, cosine, calibrate) is what lasts. 

Wrapping up

So, in spite of everything that. Video similarity is a tug-of-war between velocity, accuracy and price, and nothing wins all three without delay. 

Hashing is prompt and shallow. The CV mix is clear however miscalibrated. The large LLMs are insightful however flaky. Body embeddings sit within the candy spot. 

On my intentionally brutal all-waterfall set, Gemini Embedding 2 tracked a five-method consensus at 93% and stayed quick sufficient to cover. That’s why I shipped it. 

Three issues I’d truly inform you. Benchmark by yourself footage, not somebody’s weblog desk. Calibrate to the mannequin you truly picked. And hold the entire thing unfastened sufficient to tear the mannequin out when the following one lands. Which it can, most likely proper after you end studying this. 

Incessantly Requested Questions