Is there a limit on how many URLs I can scrape in one run?

The bulk file upload tab caps a run at 100 URLs via a slider (default 20). The paste a URL list tab has no cap, so use that for larger batches, keeping your Firecrawl plan limits and the request delay in mind.

What drives the cost?

Every URL is one Firecrawl /v1/scrape call billed to your Firecrawl account, so cost scales directly with URL count. On 429 rate limit responses it retries the same URL up to three times with 10, 20 and 40 second waits before giving up.

Why does my ZIP contain fewer files than the number of successful scrapes?

Filenames are built from the URL path with slashes replaced by underscores and truncated to 100 characters. Two URLs that normalise to the same name, or very long paths sharing their first 100 characters, overwrite each other in the archive, so the last one scraped wins.

Content is missing from JavaScript heavy pages. What do I change?

Every request sends a waitFor value (default 2,000 ms, slider up to 10,000) telling Firecrawl how long to wait for dynamic content, plus a 30 second scrape timeout. Raise the wait time if late loading content is being cut off.

Back to Tools

Firecrawl Markdown Scraper

Use cases

Content migration to new CMS LLM training data collection Competitor content archival Documentation generation

Firecrawl /v1/scrape with bearer token auth.

Retry logic with exponential backoff (max 3 attempts) for rate limiting (429).

Options: strip navigation, remove ads, strip base64 images.

Optional YAML metadata headers (URL, title, description).

ZIP archive download for bulk.

Streamlit App

Platform

Browser-based (no installation required)

Input

Firecrawl API key

URLs: single, pasted list, or CSV/Excel

Output

Markdown files with optional YAML headers. Bulk: ZIP archive + CSV summary of results.

Launch App View Source

Features

Exponential backoff retry (max 3 attempts)
Strip navigation, ads, base64 images
Optional YAML metadata headers
ZIP archive for bulk downloads
CSV summary of scraping results

How to use

1 Enter Firecrawl API key
2 Configure options (metadata, stripping)
3 Provide URLs (single, list, or file)
4 Set request delay for bulk
5 Download individual markdown, combined, or ZIP

Frequently asked questions

Is there a limit on how many URLs I can scrape in one run?: The bulk file upload tab caps a run at 100 URLs via a slider (default 20). The paste a URL list tab has no cap, so use that for larger batches, keeping your Firecrawl plan limits and the request delay in mind.
What drives the cost?: Every URL is one Firecrawl /v1/scrape call billed to your Firecrawl account, so cost scales directly with URL count. On 429 rate limit responses it retries the same URL up to three times with 10, 20 and 40 second waits before giving up.
Why does my ZIP contain fewer files than the number of successful scrapes?: Filenames are built from the URL path with slashes replaced by underscores and truncated to 100 characters. Two URLs that normalise to the same name, or very long paths sharing their first 100 characters, overwrite each other in the archive, so the last one scraped wins.
Content is missing from JavaScript heavy pages. What do I change?: Every request sends a waitFor value (default 2,000 ms, slider up to 10,000) telling Firecrawl how long to wait for dynamic content, plus a 30 second scrape timeout. Raise the wait time if late loading content is being cut off.

Want me to run this for you?

I run this tool as a managed service, or build something custom around your data. You get the insights without touching the code.

Tell Me What You Need

Schema Markup GeneratorAppTechnical SEO

Generate JSON-LD structured data for rich snippets without any API required.

Regex Generator for SEOAppTechnical SEO

Generate regex patterns from plain English descriptions for SEO tasks.

Template FingerprintingAppTechnical SEO

Identify page templates by analyzing HTML structure patterns.

Archive.org Broken Link MapperScriptTechnical SEO

Find lost URLs via Archive.org and auto-map redirects using fuzzy matching.

LLM Sitemap CreatorAppTechnical SEO

Use GPT to generate hierarchical sitemap structures from keywords.

CPC CalculatorBrowserPPC

Calculate CPC, total spend, or clicks from any two values.

Need something built for your business?

This tool started as bespoke client work. I build custom scripts, data pipelines, and full apps for SEO and product data problems that off-the-shelf tools don't solve.