Do I need an API key, and why do large domains take so long?

No key is needed: the tool queries the Wayback Machine's public CDX server directly. Results are paginated at 50,000 records per page and the tool sleeps 1 second between pages (5 seconds after an error), so a domain with millions of captures can take several minutes to fetch. Progress is shown as pages complete.

How does the tool decide a page has 'changed'?

Each Wayback capture carries a digest, a hash of the page content. The Frequently Changed Pages tab counts unique digests per URL, so a page only counts as changed when the archive captured a version with different content, not every time it was recrawled.

Why are some pages missing when I pick the 'HTML only' filter?

The filter is extension based: it keeps only URLs ending in .html, .htm, a trailing slash, or robots.txt. Extensionless URLs such as /about or /products/widget are excluded even though they are HTML pages. If your site uses extensionless URLs, export with 'All' and filter the CSV yourself.

Why does the exported CSV contain the same URL many times?

By default the export writes one row per capture, so a URL archived 200 times appears 200 times with different timestamps, status codes and digests. Tick 'Export only unique URLs' to keep a single row per URL (the first capture encountered). Columns are URL, Timestamp, Status Code and Digest, saved as UTF-8 with BOM so it opens cleanly in Excel.

Which robots.txt versions can I compare?

The robots.txt tab queries the CDX server filtered to status 200 captures and collapsed by digest, so you only see genuinely distinct versions of the file. Any two versions can be diffed side by side using Google's diff-match-patch, with the full text of both versions displayed.

Internet Archive Analyser: free Migration tool

#about

Queries the Wayback Machine CDX server to analyse how a site evolved over time.

Tracks folder structure changes, HTTP status codes, frequently modified pages, and robots.txt history with diff-based version comparison.

Includes Plotly visualisations and CSV export.

Streamlit app

Platform: Browser-based (no installation required)
Input: Domain nameAnalysis settings
Output: Historical analysis with charts and CSV export
Category: Migration

Use cases

Planning migrations by understanding old site structure
Recovering lost URLs and content from archives
Analyzing competitor site evolution over time
Finding historical content for link reclamation

#features

Folder structure evolution tracking (annual breakdown)
HTTP status code analysis (1xx-5xx groupings over time)
Frequently changed pages identified by unique content digests
robots.txt change timeline with side-by-side version comparison
Stacked line/bar chart visualisations (Plotly)
Download filters: all files, HTML only, HTML + Images, unique URLs only

#how-to-use

Enter the domain you want to analyse
Select visualisation type and top folders count
Fetch URLs from the Wayback Machine CDX server (paginated, 1s delay per page)
Explore tabs: Folder Structure, Status Codes, Changed Pages, robots.txt
Compare any two unique robots.txt versions side by side
Filter the URL list and export as CSV (per capture, or unique URLs only)

#faq

Do I need an API key, and why do large domains take so long?: No key is needed: the tool queries the Wayback Machine's public CDX server directly. Results are paginated at 50,000 records per page and the tool sleeps 1 second between pages (5 seconds after an error), so a domain with millions of captures can take several minutes to fetch. Progress is shown as pages complete.
How does the tool decide a page has 'changed'?: Each Wayback capture carries a digest, a hash of the page content. The Frequently Changed Pages tab counts unique digests per URL, so a page only counts as changed when the archive captured a version with different content, not every time it was recrawled.
Why are some pages missing when I pick the 'HTML only' filter?: The filter is extension based: it keeps only URLs ending in .html, .htm, a trailing slash, or robots.txt. Extensionless URLs such as /about or /products/widget are excluded even though they are HTML pages. If your site uses extensionless URLs, export with 'All' and filter the CSV yourself.
Why does the exported CSV contain the same URL many times?: By default the export writes one row per capture, so a URL archived 200 times appears 200 times with different timestamps, status codes and digests. Tick 'Export only unique URLs' to keep a single row per URL (the first capture encountered). Columns are URL, Timestamp, Status Code and Digest, saved as UTF-8 with BOM so it opens cleanly in Excel.
Which robots.txt versions can I compare?: The robots.txt tab queries the CDX server filtered to status 200 captures and collapsed by digest, so you only see genuinely distinct versions of the file. Any two versions can be diffed side by side using Google's diff-match-patch, with the full text of both versions displayed.

/contact

Want me to run this for you? I run this tool as a managed service, or build something custom around your data. You get the insights without touching the code.

Book a 30 minute call → [email protected]