Do I need an API key, and why do large domains take so long?

No key is needed: the tool queries the Wayback Machine's public CDX server directly. Results are paginated at 50,000 records per page and the tool sleeps 1 second between pages (5 seconds after an error), so a domain with millions of captures can take several minutes to fetch. Progress is shown as pages complete.

How does the tool decide a page has 'changed'?

Each Wayback capture carries a digest, a hash of the page content. The Frequently Changed Pages tab counts unique digests per URL, so a page only counts as changed when the archive captured a version with different content, not every time it was recrawled.

Why are some pages missing when I pick the 'HTML only' filter?

The filter is extension based: it keeps only URLs ending in .html, .htm, a trailing slash, or robots.txt. Extensionless URLs such as /about or /products/widget are excluded even though they are HTML pages. If your site uses extensionless URLs, export with 'All' and filter the CSV yourself.

Why does the exported CSV contain the same URL many times?

By default the export writes one row per capture, so a URL archived 200 times appears 200 times with different timestamps, status codes and digests. Tick 'Export only unique URLs' to keep a single row per URL (the first capture encountered). Columns are URL, Timestamp, Status Code and Digest, saved as UTF-8 with BOM so it opens cleanly in Excel.

Which robots.txt versions can I compare?

The robots.txt tab queries the CDX server filtered to status 200 captures and collapsed by digest, so you only see genuinely distinct versions of the file. Any two versions can be diffed side by side using Google's diff-match-patch, with the full text of both versions displayed.

Back to Tools

Internet Archive Analyser

Use cases

Planning migrations by understanding old site structure Recovering lost URLs and content from archives Analyzing competitor site evolution over time Finding historical content for link reclamation

Queries the Wayback Machine CDX server to analyse how a site evolved over time.

Tracks folder structure changes, HTTP status codes, frequently modified pages, and robots.txt history with diff-based version comparison.

Includes Plotly visualisations and CSV export.

Streamlit App

Platform

Browser-based (no installation required)

Input

Domain name

Analysis settings

Output

Historical analysis with charts and CSV export

Launch App View Source

Features

Folder structure evolution tracking (annual breakdown)
HTTP status code analysis (1xx-5xx groupings over time)
Frequently changed pages identified by unique content digests
robots.txt change timeline with side-by-side version comparison
Stacked line/bar chart visualisations (Plotly)
Download filters: all files, HTML only, HTML + Images, unique URLs only

How to use

1 Enter the domain you want to analyse
2 Select visualisation type and top folders count
3 Fetch URLs from the Wayback Machine CDX server (paginated, 1s delay per page)
4 Explore tabs: Folder Structure, Status Codes, Changed Pages, robots.txt
5 Compare any two unique robots.txt versions side by side
6 Filter the URL list and export as CSV (per capture, or unique URLs only)

Frequently asked questions

Do I need an API key, and why do large domains take so long?: No key is needed: the tool queries the Wayback Machine's public CDX server directly. Results are paginated at 50,000 records per page and the tool sleeps 1 second between pages (5 seconds after an error), so a domain with millions of captures can take several minutes to fetch. Progress is shown as pages complete.
How does the tool decide a page has 'changed'?: Each Wayback capture carries a digest, a hash of the page content. The Frequently Changed Pages tab counts unique digests per URL, so a page only counts as changed when the archive captured a version with different content, not every time it was recrawled.
Why are some pages missing when I pick the 'HTML only' filter?: The filter is extension based: it keeps only URLs ending in .html, .htm, a trailing slash, or robots.txt. Extensionless URLs such as /about or /products/widget are excluded even though they are HTML pages. If your site uses extensionless URLs, export with 'All' and filter the CSV yourself.
Why does the exported CSV contain the same URL many times?: By default the export writes one row per capture, so a URL archived 200 times appears 200 times with different timestamps, status codes and digests. Tick 'Export only unique URLs' to keep a single row per URL (the first capture encountered). Columns are URL, Timestamp, Status Code and Digest, saved as UTF-8 with BOM so it opens cleanly in Excel.
Which robots.txt versions can I compare?: The robots.txt tab queries the CDX server filtered to status 200 captures and collapsed by digest, so you only see genuinely distinct versions of the file. Any two versions can be diffed side by side using Google's diff-match-patch, with the full text of both versions displayed.

Want me to run this for you?

I run this tool as a managed service, or build something custom around your data. You get the insights without touching the code.

Book a Call

Website Migration ToolAppMigration

Automate redirect mapping during migrations using PolyFuzz string matching.

Keyword Difficulty CheckerAppKeyword Research

Check keyword difficulty and search intent using the DataForSEO Labs API.

PDF Branding InjectorAppE-commerce

Add branded headers to PDFs using PyPDF2 and ReportLab with responsive text scaling.

SERP Title Keyword ExtractorAppKeyword Research

Extract keyword fragments from organic result titles with FuzzyWuzzy similarity scoring and Altair frequency charts.

Product Title OptimiserScriptE-commerce

LLM-powered title restructuring with numerical validation and 80% word overlap checks.

Break-even ROAS CalculatorBrowserPPC

Calculate the minimum ROAS needed to break even.

Need something built for your business?

This tool started as bespoke client work. I build custom scripts, data pipelines, and full apps for SEO and product data problems that off-the-shelf tools don't solve.