What exactly must the uploaded CSV contain?

Three columns with these exact names: Address, H1-1 and Copy 1. Address and H1-1 come from a standard Screaming Frog internal HTML export; Copy 1 must come from a custom extraction you set up before crawling to capture the main page copy. Rows with an empty Copy 1 are dropped before analysis (the app tells you how many), and at least 2 URLs with content are required. UTF-8 is tried first with a Latin-1 fallback.

Will it catch pages that are 100% identical?

Often not, and this is the main gotcha. Matching happens on the copy text, and URLs are attached afterwards by looking the text up in a deduplicated table, so two pages with byte-identical copy resolve to the same URL and the pair is then discarded as a self-match. The tool is built for near-duplicates; find exact duplicates separately, for example by grouping the Copy 1 column on identical values (or Screaming Frog's own exact duplicates report).

What is the difference between the two similarity sliders?

Minimum Similarity Score (default 0.9) filters which pairs appear in the final results. Group Link Similarity (default 0.75) is passed to PolyFuzz's grouping step and controls how aggressively matches are chained into named clusters, which is where the Duplicate Cluster Name comes from. The cluster is named after the representative page's H1, so pages with blank H1s produce blank cluster names.

Is the similarity semantic or lexical?

Lexical. PolyFuzz's TF-IDF matcher compares the text vectors directly, so two paragraphs saying the same thing in different words score low, while pages sharing large chunks of boilerplate score high. Make sure your Copy custom extraction targets the main content only; if it captures shared template text, everything will look like a duplicate.

What does the Pages in Cluster column count?

It counts duplicate pairs (rows) sharing the same cluster name, not distinct pages, so a cluster of 3 mutually similar pages can show a higher number than the page count. Treat it as a relative size indicator for prioritising clusters.

Back to Tools

Content Duplication Finder

Use cases

Finding duplicate product descriptions Identifying thin content to consolidate Content audit for large sites E-commerce duplicate content cleanup

Uses PolyFuzz TF-IDF vectorisation for similarity matching with hierarchical clustering to group related content.

Three configurable thresholds: minimum similarity score (0.5-1.0), URL filter pattern, and group link similarity (0.5-1.0) for cluster formation.

Requires Screaming Frog CSV with Address, H1-1, and Copy 1 columns.

Minimum 2 URLs required.

Streamlit App Crawl Data

Platform

Browser-based (no installation required)

Input

Screaming Frog CSV with columns: Address, H1-1, Copy 1

Minimum 2 URLs with content required

Output

CSV with duplicate pairs and clusters

Launch App View Source

Features

PolyFuzz TF-IDF vectorisation
Hierarchical clustering for duplicate grouping
Minimum similarity score slider (0.5-1.0)
Group link similarity threshold (0.5-1.0)
URL filter for targeting specific sections
UTF-8 and Latin-1 encoding support

How to use

1 Crawl your site with Screaming Frog (enable custom extraction for Copy 1)
2 Export internal_html.csv
3 Upload and set minimum similarity score (0.9 = 90% threshold)
4 Optionally filter by URL pattern
5 Adjust group link similarity for cluster tightness
6 Download CSV with duplicate clusters

Frequently asked questions

What exactly must the uploaded CSV contain?: Three columns with these exact names: Address, H1-1 and Copy 1. Address and H1-1 come from a standard Screaming Frog internal HTML export; Copy 1 must come from a custom extraction you set up before crawling to capture the main page copy. Rows with an empty Copy 1 are dropped before analysis (the app tells you how many), and at least 2 URLs with content are required. UTF-8 is tried first with a Latin-1 fallback.
Will it catch pages that are 100% identical?: Often not, and this is the main gotcha. Matching happens on the copy text, and URLs are attached afterwards by looking the text up in a deduplicated table, so two pages with byte-identical copy resolve to the same URL and the pair is then discarded as a self-match. The tool is built for near-duplicates; find exact duplicates separately, for example by grouping the Copy 1 column on identical values (or Screaming Frog's own exact duplicates report).
What is the difference between the two similarity sliders?: Minimum Similarity Score (default 0.9) filters which pairs appear in the final results. Group Link Similarity (default 0.75) is passed to PolyFuzz's grouping step and controls how aggressively matches are chained into named clusters, which is where the Duplicate Cluster Name comes from. The cluster is named after the representative page's H1, so pages with blank H1s produce blank cluster names.
Is the similarity semantic or lexical?: Lexical. PolyFuzz's TF-IDF matcher compares the text vectors directly, so two paragraphs saying the same thing in different words score low, while pages sharing large chunks of boilerplate score high. Make sure your Copy custom extraction targets the main content only; if it captures shared template text, everything will look like a duplicate.
What does the Pages in Cluster column count?: It counts duplicate pairs (rows) sharing the same cluster name, not distinct pages, so a cluster of 3 mutually similar pages can show a higher number than the page count. Treat it as a relative size indicator for prioritising clusters.

Want me to run this for you?

I run this tool as a managed service, or build something custom around your data. You get the insights without touching the code.

Tell Me What You Need

Content ExtractorAppContent

Extract main text content and H1 headings from URLs.

Reading Score AnalyserAppContent

Analyse content readability from sitemaps or URLs using Flesch scores.

Page Intent ClassifierAppContent

Use OpenAI to classify page intent and expected user actions.

Review Sentiment ExtractorAppContent

Use OpenAI to extract positive and negative sentiments from product reviews.

Keyword-to-Page MapperAppContent

Semantically match keywords to existing pages using ML embeddings.

Redirect List FormatterBrowserMigration

Convert URL lists into redirect rules for different platforms.

Need something built for your business?

This tool started as bespoke client work. I build custom scripts, data pipelines, and full apps for SEO and product data problems that off-the-shelf tools don't solve.