Archive.org Broken Link Mapper

Features

• Wayback Machine CDX API bulk URL download
• Filters to text/html; removes static assets, parameters, and tracking URLs
• Excludes URLs already present in the live crawl
• waybackpy newest-snapshot fetching via ThreadPoolExecutor (default 15 threads)
• BeautifulSoup H1 extraction from archived pages
• PolyFuzz TF-IDF H1-to-H1 redirect matching
• Optional live HTTP status check with 429 retry/backoff
• Safety backup CSV written before status checking

How to use

1. 1 Place internal_html.csv in the script folder
2. 2 Configure threads (8-10 recommended), user agent, and check_status flag
3. 3 Run the script; the domain is auto-detected from the crawl
4. 4 Review redirect suggestions sorted by similarity
5. 5 Use urls-to-redirect-archive-org.csv for redirect implementation

Frequently asked questions

What exactly does the input file need to be?

A file named internal_html.csv in the same folder as the script, with Address and H1-1 columns exactly as Screaming Frog names them. Rows with a blank H1-1 are dropped, and the target domain is auto detected from the first Address row, so make sure row one is on the domain you want to recover.

Why are some genuinely lost URLs missing from the output?

The still-live check uses substring matching: an archived URL is discarded if any live crawl URL appears anywhere inside it. If your crawl contains /shoes, an archived /shoes-sale is discarded too, even though it no longer exists. URLs with parameters or fragments and static asset extensions are also filtered out before matching.

How much should I trust the redirect suggestions?

Matching pairs the newest Wayback snapshot's H1 with the closest live crawl H1 using PolyFuzz TF-IDF, and there is no minimum similarity threshold in the code: every archived page gets its best available match. The output is sorted by similarity, so set your own cutoff and manually review anything below it.

Which archived pages get silently excluded?

Snapshots where no H1 could be read are recorded as "No Data Received!" and dropped before matching, and snapshots flagged by the Wayback Machine as "Got an HTTP 301 response at crawl time" are also removed.

Should I turn on check_status?

It is off by default. When enabled the script makes a live GET request to every matched source URL, with up to five retries and a long backoff on 429 responses, which is slow on large lists. A safety backup CSV is always written before the status check starts, so you can also verify statuses separately later.