Where do I put my URLs and API key?

Everything is configured by editing the top of the script, there are no command line flags. Set INPUT_FILE to a plain text file with one URL per line, set OUTPUT_DIR for the results, and paste your Anthropic API key into the api_key variable. The shipped paths are Windows-style (C:\python_scripts\...), so on Mac or Linux you must change them before the first run.

Which Claude model does it use and what does it cost?

It calls Claude Haiku 4.5 (claude-haiku-4-5-20251001) with a 4,000 output token cap and prompt caching enabled, one API call per URL. The readme estimates $0.002 to $0.015 per URL depending on page size, and the console prints input, cache and output token counts after every request so you can track spend. Set DEBUG_MODE = True to trial the first 2 URLs before committing to a full run.

Why are my frequency counts higher than the number of URLs I processed?

The post-processing step combines every content_blocks_*.csv it finds in the output folder, including files left over from previous runs, before counting XPath frequencies. Clear or archive old batch files from the output directory between runs or the counts in combined_output_with_frequency.csv will be inflated.

How is the frequency column calculated?

By exact string match on the XPath. Two pages only count towards the same pattern if Claude returned character-identical XPath expressions for both, so near-identical selectors (a different class order, an added index) are counted separately. Block names are standardised afterwards: every row sharing an XPath gets the first name that appeared for it.

What happens if a URL fails mid-run?

Fetch errors and unparseable Claude responses are logged and the URL is skipped, the run continues. Results are written to disk in batches of 50 rows, so a crash loses at most the rows since the last incremental save. There is a fixed 1 second delay before each page fetch and a 30 second request timeout.

Back to Tools

Content Block Extractor

Use cases

Identifying common template patterns at scale Finding content blocks for scraping or migration Analysing page structure consistency Generating XPaths for content extraction

Uses Claude Haiku to analyse filtered HTML and identify major content blocks with XPath expressions.

Removes script, style, noscript, meta, link, header, footer, and nav tags to reduce token usage.

Aggregates results across URLs to find common XPath patterns with frequency counts.

Requires API Key

Platform

Python script (requires Python 3.x)

Input

URL list

Claude API key

Output

CSV with content blocks and XPath patterns

View Source

Features

Claude Haiku API for block detection
XPath expression generation for each block
HTML filtering removes nav/header/footer/scripts
Frequency analysis of XPath patterns across pages
Incremental CSV saves every 50 rows
Token usage reporting (input, cache, output)

How to use

1 Add URLs to input/urls.txt file
2 Configure Claude API key
3 Run script (1-second delay between requests)
4 Review combined_output_with_frequency.csv
5 Analyse top 10 XPath patterns by frequency

Frequently asked questions

Where do I put my URLs and API key?: Everything is configured by editing the top of the script, there are no command line flags. Set INPUT_FILE to a plain text file with one URL per line, set OUTPUT_DIR for the results, and paste your Anthropic API key into the api_key variable. The shipped paths are Windows-style (C:\python_scripts\...), so on Mac or Linux you must change them before the first run.
Which Claude model does it use and what does it cost?: It calls Claude Haiku 4.5 (claude-haiku-4-5-20251001) with a 4,000 output token cap and prompt caching enabled, one API call per URL. The readme estimates $0.002 to $0.015 per URL depending on page size, and the console prints input, cache and output token counts after every request so you can track spend. Set DEBUG_MODE = True to trial the first 2 URLs before committing to a full run.
Why are my frequency counts higher than the number of URLs I processed?: The post-processing step combines every content_blocks_*.csv it finds in the output folder, including files left over from previous runs, before counting XPath frequencies. Clear or archive old batch files from the output directory between runs or the counts in combined_output_with_frequency.csv will be inflated.
How is the frequency column calculated?: By exact string match on the XPath. Two pages only count towards the same pattern if Claude returned character-identical XPath expressions for both, so near-identical selectors (a different class order, an added index) are counted separately. Block names are standardised afterwards: every row sharing an XPath gets the first name that appeared for it.
What happens if a URL fails mid-run?: Fetch errors and unparseable Claude responses are logged and the URL is skipped, the run continues. Results are written to disk in batches of 50 rows, so a crash loses at most the rows since the last incremental save. There is a fixed 1 second delay before each page fetch and a 30 second request timeout.

Want me to run this for you?

I run this tool as a managed service, or build something custom around your data. You get the insights without touching the code.

Tell Me What You Need

Content Consolidation AnalyserScriptContent

Find cannibalising pages by clustering queries that share SERP URLs.

Content Hub ClassifierAppContent

Classify articles into content hubs using OpenAI models with structured JSON output.

Entity ExtractorAppContent

Extract named entities from content for semantic SEO analysis.

AI Entity VisualiserAppContent

Extract and visualise named entities using GPT with D3.js circle packing.

SERP N-gram ExtractorScriptContent

Extract n-grams and title keywords from SERP content for content optimisation.

Redirect List FormatterBrowserMigration

Convert URL lists into redirect rules for different platforms.

Need something built for your business?

This tool started as bespoke client work. I build custom scripts, data pipelines, and full apps for SEO and product data problems that off-the-shelf tools don't solve.