Where do I get the API token?

Log in to app.oncrawl.com, click your profile icon then Account Settings, scroll to API Access Tokens and generate or copy a token. Paste it into the sidebar; it is used as a bearer token for every request.

Why does my crawl not appear in the dropdown?

Data Extraction, Aggregations, Crawl Comparison and Link Export only list crawls with status "done". Running, paused or failed crawls can be managed in the Crawl Management module but cannot be queried. Lookups are also cached for 5 minutes, so a crawl that just finished may take a few minutes to show up.

Do preset queries cover every crawled URL?

No. Every preset filters to fetched pages, and most content, canonical and link presets additionally require status 200, so for example Pages Without Title or Orphan Pages only report 200 pages and exclude redirects and errors. All Fetched Pages is the preset without a status filter.

Are there export size limits?

Full exports call OnCrawl's export endpoint with a 300 second timeout, so very large result sets can time out. Use the optional URL filter (a contains match on the URL) to split big extractions into smaller chunks. OnCrawl returns semicolon separated CSV; the app detects semicolon versus comma automatically before parsing.

What can aggregations do?

One grouping field per query (status code, depth, URL segments, content type, canonical evaluation and a few others) with a URL count per bucket, optionally filtered by the same OQL as the presets. It is a count only aggregation; there are no sums or averages.

Back to Tools

OnCrawl API Suite

Use cases

Technical SEO audits - 35+ issue presets Crawl automation - launch and schedule crawls Change tracking - compare crawls over time Link analysis - export internal/external links Project management - create projects via API

OnCrawl API v2 with bearer token authentication.

7 modules: Data Extraction (35+ preset OQL queries), Crawl Management (launch/pause/resume/cancel), Aggregations (groupBy queries), Crawl Comparison (new/lost/changed pages), Project CRUD, Scheduling (daily/weekly/monthly), Link Export.

60+ data fields in 14 categories.

TTL caching.

Streamlit App

Platform

Browser-based (no installation required)

Input

OnCrawl API token (bearer authentication)

Workspace and project selection

Crawl selection for queries

Output

Crawl data with intelligent CSV parsing. Aggregation results with count sorting. Comparison exports with page status (new/lost/changed).

Launch App View Source

Features

35+ preset OQL queries with custom builder
Crawl lifecycle: launch, pause, resume, cancel
Crawl comparison: new, lost, changed pages
Aggregation normalisation (cols/rows to buckets)
Schedule creation: daily/weekly/monthly
60+ data fields across 14 categories

How to use

1 Enter your OnCrawl API token
2 Select workspace and project from dropdowns
3 Choose a module: Data Extraction, Crawl Management, Aggregations, Comparison, Projects, Scheduling, or Link Export
4 Use presets or build a custom OQL query
5 Download CSV (auto delimiter detection) or Excel

Frequently asked questions

Where do I get the API token?: Log in to app.oncrawl.com, click your profile icon then Account Settings, scroll to API Access Tokens and generate or copy a token. Paste it into the sidebar; it is used as a bearer token for every request.
Why does my crawl not appear in the dropdown?: Data Extraction, Aggregations, Crawl Comparison and Link Export only list crawls with status "done". Running, paused or failed crawls can be managed in the Crawl Management module but cannot be queried. Lookups are also cached for 5 minutes, so a crawl that just finished may take a few minutes to show up.
Do preset queries cover every crawled URL?: No. Every preset filters to fetched pages, and most content, canonical and link presets additionally require status 200, so for example Pages Without Title or Orphan Pages only report 200 pages and exclude redirects and errors. All Fetched Pages is the preset without a status filter.
Are there export size limits?: Full exports call OnCrawl's export endpoint with a 300 second timeout, so very large result sets can time out. Use the optional URL filter (a contains match on the URL) to split big extractions into smaller chunks. OnCrawl returns semicolon separated CSV; the app detects semicolon versus comma automatically before parsing.
What can aggregations do?: One grouping field per query (status code, depth, URL segments, content type, canonical evaluation and a few others) with a URL count per bucket, optionally filtered by the same OQL as the presets. It is a count only aggregation; there are no sums or averages.

Want me to run this for you?

I run this tool as a managed service, or build something custom around your data. You get the insights without touching the code.

Tell Me What You Need

Q&A Schema ExtractorAppTechnical SEO

Extract Question/Answer pairs from JSON-LD schema markup.

Firecrawl Markdown ScraperAppTechnical SEO

Convert web pages to clean markdown using Firecrawl API.

Schema Markup GeneratorAppTechnical SEO

Generate JSON-LD structured data for rich snippets without any API required.

Regex Generator for SEOAppTechnical SEO

Generate regex patterns from plain English descriptions for SEO tasks.

Template FingerprintingAppTechnical SEO

Identify page templates by analyzing HTML structure patterns.

Schema Markup GeneratorBrowserTechnical SEO

Generate JSON-LD structured data for common schema types.

Need something built for your business?

This tool started as bespoke client work. I build custom scripts, data pipelines, and full apps for SEO and product data problems that off-the-shelf tools don't solve.