Why are the column dropdowns empty after uploading my file?

The dropdowns only offer columns with expected names. The keyword column must be named KW, kw, Keyword, keyword, keywords or Keywords, and the volume column must be vol, Vol, Volume, volume, search volume or Search Volume. Rename your headers to one of these before uploading.

Which of two near-duplicate keywords gets kept?

The one that appears first in your file, not the one with the highest volume. The volume column is selected in the UI but never used in the keep/drop decision, so sort your file by search volume descending before uploading if you want the top-volume variant of each duplicate group to survive.

What actually counts as a near duplicate?

Matching uses rapidfuzz token_sort_ratio with a fixed 99 threshold that cannot be changed in the UI. In practice that catches word reorderings ('mens running shoes' vs 'running shoes mens') and near-identical strings, while plurals and small word substitutions usually score below 99 and are kept as separate keywords.

Back to Tools

Keyword Deduplication Tool

Use cases

Cleaning keyword lists before clustering Removing near-duplicate keywords from exports Consolidating keyword research from multiple sources

Uses rapidfuzz with token_sort_ratio scorer and process.extractOne() to identify near-duplicate keywords at 99-point similarity threshold.

Automatically detects file encoding via chardet (first 100,000 bytes).

Keeps the first occurrence in file order (sort by volume descending before upload to keep the highest volume variant) and exports both processed and dropped keywords for review.

Uses stqdm for progress tracking.

Streamlit App

Platform

Browser-based (no installation required)

Input

CSV or Excel file with keywords and search volumes

Encoding auto-detected via chardet

Output

Excel with deduplicated and dropped keywords

Launch App View Source

Features

Rapidfuzz token_sort_ratio with a fixed 99-point threshold
Chardet encoding detection (100KB sample)
Keeps the first occurrence of each duplicate group (sort by volume first to keep top-volume variants)
Column dropdowns require preset header names (Keyword/KW and Volume/vol variants)
stqdm progress bar integration
Two-sheet Excel output via xlsxwriter
Supports CSV and Excel (.xlsx) input

How to use

1 Name your columns Keyword and Volume (or another preset variant) before upload
2 Sort the file by search volume descending so the highest volume variants are kept
3 Upload the CSV or Excel file
4 Select the keyword and volume columns from the dropdowns
5 Click Dedupe to run rapidfuzz matching
6 Download Excel with Processed Keywords and Dropped Keywords sheets

Frequently asked questions

Why are the column dropdowns empty after uploading my file?: The dropdowns only offer columns with expected names. The keyword column must be named KW, kw, Keyword, keyword, keywords or Keywords, and the volume column must be vol, Vol, Volume, volume, search volume or Search Volume. Rename your headers to one of these before uploading.
Which of two near-duplicate keywords gets kept?: The one that appears first in your file, not the one with the highest volume. The volume column is selected in the UI but never used in the keep/drop decision, so sort your file by search volume descending before uploading if you want the top-volume variant of each duplicate group to survive.
What actually counts as a near duplicate?: Matching uses rapidfuzz token_sort_ratio with a fixed 99 threshold that cannot be changed in the UI. In practice that catches word reorderings ('mens running shoes' vs 'running shoes mens') and near-identical strings, while plurals and small word substitutions usually score below 99 and are kept as separate keywords.

Want me to run this for you?

I run this tool as a managed service, or build something custom around your data. You get the insights without touching the code.

Tell Me What You Need

AdWords Tool SetAppKeyword Research

Extract MPNs, keyphrases, and calculate optimal bid adjustments from Google Ads data.

Keyword GrouperAppKeyword Research

Group similar keywords using PolyFuzz TF-IDF fuzzy matching at scale.

Category Keyword FinderAppKeyword Research

Discover new category opportunities by extracting n-grams from product titles.

Keyword Trends AnalyserAppKeyword Research

Analyse keyword trends using Google Trends to find rising and declining topics.

Topical Map GeneratorAppKeyword Research

Use AI (GPT-4o) to organise keywords into hierarchical topical maps.

ROAS CalculatorBrowserPPC

Calculate Return on Ad Spend from revenue and ad costs.

Need something built for your business?

This tool started as bespoke client work. I build custom scripts, data pipelines, and full apps for SEO and product data problems that off-the-shelf tools don't solve.