Template Fingerprinting

Features

• TF-IDF vectorisation of HTML structural features
• K-Means clustering (configurable cluster count, default 5)
• Four feature dimensions: tag counts, CSS classes, IDs, meta tags
• Reproducible results (random state 42)
• Bulk URL fetching with progress indicator

How to use

1. 1 Upload a CSV of URLs (Address column preferred; URL/url/address also accepted, else the first column is used)
2. 2 Set the number of template types (2-20, default 5) and request timeout
3. 3 Click Analyze Templates; each URL is fetched live and clustered
4. 4 Review assignments, distribution charts, and per-cluster top features with sample URLs
5. 5 Download CSV with URL, Cluster, and Page Type columns

Frequently asked questions

Does it analyse HTML from my crawl file or fetch pages itself?

It fetches every URL live at analysis time; the CSV only supplies the URL list. Each page is requested with a 10 second default timeout and a short delay between requests, so a large list takes a while and puts one request per URL on the target server.

What does the input CSV need?

An Address column is expected, but the app also accepts URL, url, address or Url, and otherwise falls back to the first column with a warning. Only values starting with http are kept, so relative paths are silently discarded.

What happens to URLs that fail to fetch?

In the Streamlit app they are excluded from clustering and listed in a failed URLs expander. In the command line script a failed fetch becomes an empty feature string that still gets clustered, so a cluster of fetch failures can masquerade as a template type. The script also sends no custom user agent (the python-requests default), making blocks more likely than in the app, which sends a Mozilla user agent.

How do I work out what each cluster actually is?

Clusters are only labelled Type 0, Type 1 and so on. The app's Cluster Details tab shows the top ten distinguishing structural features and five sample URLs per cluster (the script prints the top five features to the console); you inspect those and name the templates yourself.

Will I get the same clusters if I rerun it?

For identical input, yes: K-Means runs with a fixed random state (42), so the same URL set and cluster count give the same assignments. Adding or removing URLs, or changing the cluster count, can reshuffle all cluster numbers.

Does it see JavaScript rendered markup?

No. Features come from the raw HTML response fetched with the requests library, so structure injected client side is invisible. Two templates that only differ after rendering can end up in the same cluster.