How does the tool decide which CSS selector contains the main content?

It strips non-content elements (nav, header, footer, aside, script, style, plus elements whose class or ID contains nav/menu/sidebar/footer/header), then generates structural summaries of the top-level containers (selector path, text length, and a 200-character sample). These summaries are sent to the LLM with a system prompt asking it to return the best selector as JSON. Only the top 5 containers are included in the prompt.

What does the 'specific selector' refinement do?

After the LLM returns an initial selector, the tool walks down the DOM from that element, at each level picking the child whose text length multiplied by its share of the parent's text is highest. It appends each child's tag name to the selector path until no further refinement improves specificity. This targets the most content-dense descendant rather than a broad wrapper.

How is duplicate content handled in the extracted text?

After converting to Markdown and stripping links and images, paragraphs are compared pairwise using difflib SequenceMatcher. Any paragraph with a similarity ratio above 0.8 against an already-kept paragraph is dropped. This removes near-identical boilerplate but can also discard legitimately repeated content.

Can I use a local LLM instead of OpenAI?

Yes. The tool uses the OpenAI client library, so any server exposing an OpenAI-compatible /v1/chat/completions endpoint works. Change the Base URL field in the sidebar to your local server (e.g. http://localhost:1234/v1 for LM Studio) and enter any non-empty string as the API key. The model must support JSON response format.

What happens if the LLM picks the wrong selector?

The extracted content will come from the wrong area of the page. The tool does not validate the LLM's choice against any ground truth. However, even if the selector is wrong, the fallback content extraction (find_main_content) independently scores all div, article, section, and main elements by word count weighted against link density, so the extracted text may still be reasonable.

Back to Tools

Auto CSS Selector Detector

Use cases

Extracting main content from pages without knowing the CSS structure Building content extraction pipelines for unfamiliar sites Identifying the correct selector for scraping or migration tools Converting page content to clean Markdown for analysis

Fetches a URL, removes non-content elements (nav, header, footer, aside, script, style), generates structural summaries of top-level containers sorted by a content score (text length, text-to-HTML ratio, headings, links), and sends them to an OpenAI-compatible LLM which returns the optimal CSS selector as JSON.

The selector is then refined to target the most content-rich descendant.

Content is extracted with BeautifulSoup, converted to Markdown via html2text, and deduplicated using SequenceMatcher (0.8 threshold).

Internal links are also extracted from the Markdown output.

Requires API Key

Platform

Python script (requires Python 3.x)

Input

URL to analyse

API key (OpenAI or compatible provider)

Model name (default: gpt-4o-mini)

Base URL (changeable for local LLM servers)

Output

Detected CSS selector with reasoning, refined specific selector, extracted page content as Markdown/plaintext, and a list of internal links found within the content.

View Source

Features

LLM-powered CSS selector detection via structural summaries
Selector refinement to most content-rich descendant element
Non-content stripping: nav, header, footer, aside, script, style
Class and ID filtering for menu, sidebar, footer, header patterns
Content deduplication via SequenceMatcher (0.8 similarity threshold)
Internal link extraction from converted Markdown
Supports any OpenAI-compatible API (local LLMs, OpenRouter, etc.)
Default model: gpt-4o-mini

How to use

1 Enter your API key in the sidebar
2 Optionally change the model or base URL for local LLM support
3 Paste a URL and click Detect & Extract
4 Review the detected selector and LLM reasoning
5 View the refined specific selector and internal links found
6 Download the extracted content as a .txt file

Frequently asked questions

How does the tool decide which CSS selector contains the main content?: It strips non-content elements (nav, header, footer, aside, script, style, plus elements whose class or ID contains nav/menu/sidebar/footer/header), then generates structural summaries of the top-level containers (selector path, text length, and a 200-character sample). These summaries are sent to the LLM with a system prompt asking it to return the best selector as JSON. Only the top 5 containers are included in the prompt.
What does the 'specific selector' refinement do?: After the LLM returns an initial selector, the tool walks down the DOM from that element, at each level picking the child whose text length multiplied by its share of the parent's text is highest. It appends each child's tag name to the selector path until no further refinement improves specificity. This targets the most content-dense descendant rather than a broad wrapper.
How is duplicate content handled in the extracted text?: After converting to Markdown and stripping links and images, paragraphs are compared pairwise using difflib SequenceMatcher. Any paragraph with a similarity ratio above 0.8 against an already-kept paragraph is dropped. This removes near-identical boilerplate but can also discard legitimately repeated content.
Can I use a local LLM instead of OpenAI?: Yes. The tool uses the OpenAI client library, so any server exposing an OpenAI-compatible /v1/chat/completions endpoint works. Change the Base URL field in the sidebar to your local server (e.g. http://localhost:1234/v1 for LM Studio) and enter any non-empty string as the API key. The model must support JSON response format.
What happens if the LLM picks the wrong selector?: The extracted content will come from the wrong area of the page. The tool does not validate the LLM's choice against any ground truth. However, even if the selector is wrong, the fallback content extraction (find_main_content) independently scores all div, article, section, and main elements by word count weighted against link density, so the extracted text may still be reasonable.

Want me to run this for you?

I run this tool as a managed service, or build something custom around your data. You get the insights without touching the code.

Tell Me What You Need

Review TaggerScriptContent

Tag customer reviews with a one or two-word topic label using OpenAI GPT models in batches.

Competitor Content Gap FinderScriptContent

Discover which descriptive words competitors use in titles that you are missing.

Content Block ExtractorScriptContent

Extract content blocks and XPath patterns using Claude Haiku for template analysis.

Content Consolidation AnalyserScriptContent

Find cannibalising pages by clustering queries that share SERP URLs.

Content Hub ClassifierAppContent

Classify articles into content hubs using OpenAI models with structured JSON output.

SERP Snippet PreviewBrowserContent

Preview how your page will appear in Google search results.

Need something built for your business?

This tool started as bespoke client work. I build custom scripts, data pipelines, and full apps for SEO and product data problems that off-the-shelf tools don't solve.