Embeddings for SEO: six ways cosine lies on a live corpus

Short version, for those who won’t scroll: embeddings find the duplicates and link candidates that string matching can’t see — and then mislead you in six specific, repeatable ways, each of which shipped something wrong before we caught it.

I opened the link-candidate ranking on a live corpus expecting topical neighbours, and the top pair was a legal disclaimer matched with itself — the same paragraph on two different pages, at cosine 1.0. That’s where theory and practice parted ways, so: theory first, then where it broke.

String matching has a hard ceiling: two pages can cover the same topic with almost no shared vocabulary — “cheapest eSIM for Japan” and “prepaid data plans for Tokyo travellers” overlap on roughly nothing — and every duplicate-detection tool built on word overlap will call them unrelated. Embeddings remove that ceiling, because a model turns each text into a vector positioned by meaning, and closeness becomes arithmetic.

That part is well covered elsewhere. What isn’t covered is the next part — the six specific ways cosine similarity misleads you once you point it at a real corpus instead of a demo. These are field notes, each one from a run that shipped something wrong before we caught it.

What embeddings actually buy you

Three corpus questions that are unanswerable with string matching, and trivially answerable with vectors:

• Near-duplicates that share no words. Paraphrase, translation drift, two writers briefed from the same outline.
• Latent cannibalization. Page A’s body covers page B’s topic while A claims to be about something else — see the cannibalization Ahrefs can’t see for how direction, not similarity, is the signal there.
• Internal-link candidates and orphans. Which paragraph on page A is genuinely the closest thing to page B’s topic — and which pages sit far from everything, semantically orphaned even when they have inbound links.

All three ride on one primitive, cosine similarity between vectors, which is exactly why its failure modes matter.

1. Boilerplate pairs with itself at cosine 1.0

The most expensive lesson — the one from the opening scene. A site-wide paragraph — a YMYL disclaimer, an exchange-rate note, a CTA block — is byte-identical on every page, and identical text embeds to an identical vector. So every article is maximally similar to every other article through the block that carries no topic at all, and an internal linker faithfully links on the disclaimer.

We watched this happen on a 52-page corpus: link candidates ranked by similarity, and the top of the ranking was the same legal paragraph pairing with itself, over and over. My first hypothesis was a bug in the ranking code (it usually is), so I checked the pairs by hand — and the code was right, which is somehow worse: the math had done exactly what we asked!

2. There is no universal “duplicate” threshold

Every tool that quotes a magic cosine number without naming its model and its unit of text is guessing, because the number moves with both.

On our corpora, paragraph-level paraphrase lands around 0.92 and up, so the near-duplicate cut sits at 0.90 — deliberately conservative. Internal-link candidates use 0.65. Same math, different numbers — why? Because the two questions have opposite error costs: a false duplicate verdict merges or 301s a page that should have lived, while a weak link suggestion costs a human three seconds to reject. Pick the threshold from what a mistake costs, then calibrate it on your own corpus — and a vendor who won’t name their model has probably answered the threshold question too.

3. Length skews the number

A 1,800-word body and a five-word target keyword never reach a high cosine, no matter how perfectly the page is about that keyword, because long text averages toward the centre of the space while short text sits at the edges. Compare like with like — body to body, keyword-phrase to keyword-phrase — or compare gaps between two similarities rather than the absolute value of either.

This is the single most common way a home-made semantic audit produces nonsense: it compares a page body to a query string, sees 0.58, and concludes the page is off-topic. Off-topic compared to what, exactly? Compared to a five-word string that no page on earth could sit next to.

4. Cosine does not know direction

“A is similar to B” is symmetric, and real content relationships are not. A comprehensive guide that covers a subtopic is not the same situation as a thin page that duplicates that guide’s topic — one deserves a link, the other deserves a merge, and a single similarity score cannot tell them apart. So which one is it? You need a second measurement (what each page claims to be about, versus what its body covers) and the difference between the two.

5. A cached vector describes the page you no longer have

Caching embeddings is mandatory — a re-run over an unchanged corpus should cost nothing — but the cache key has to be the text, not the URL or the slug. Key it by URL and every audit after an edit reasons about the previous version of the page, silently and forever. The symptom is nasty precisely because there isn’t one: no error, no warning, just an audit that keeps agreeing with itself, which (I’ve noticed) is often what a confident wrong audit looks like.

6. The spend is invisible, not large

Embedding a normal site costs cents. The problem isn’t the amount — it’s that the call hides inside an audit that feels free, so it never appears on a cost line until someone specifically goes hunting. We shipped three audits that embedded paid vectors without a single metering line, and the cost report happily printed a total that excluded them. Was anyone lying? No — every number in that report was correct, it just wasn’t complete, and arguably an incomplete cost report is more dangerous than a wrong one.

Meter at the shared helper where the call is actually made, never in each caller — the next audit someone adds will forget, and the guard has to make forgetting impossible.

The practical setup

If you’re adding embeddings to a content workflow, here’s the order that survives contact with a real corpus:

1. Strip repeats first, so verbatim paragraphs on 2+ pages never reach the model.
2. Cache by text hash: unchanged text, zero cost, correct vector.
3. Two thresholds, not one — conservative for destructive verdicts (merge, 301), permissive for suggestions a human reviews.
4. Compare like with like — body↔body, claim↔claim — and use gaps, not absolutes, when the lengths differ.
5. Meter at the call site — one place, so it can’t be forgotten.
6. Positive-control your zero, because “No duplicates found” and “the scan silently measured nothing” look identical in a report; plant a known duplicate and confirm the audit catches it — otherwise a clean result proves nothing.

That last one is the habit that generalises well beyond embeddings: silence is not the same as cleanliness, and any audit that can’t tell you which of the two it just produced is not finished — it’s merely quiet.

Our own semantic audits — duplicate scanning, entity shadowing, link-equity spread — run on exactly this pipeline, and each of the six rules above exists because breaking it shipped something wrong first. Is that a flattering way to build a product? Not especially, but it’s the honest one. If you want the shape of the whole system rather than the primitive, topical authority and internal-linking silos covers where these scores land in a site architecture — and if you only take one thing from here, make it rule six!