Set up Firecrawl with an AI agent

name: firecrawl
description: Use when building AI agents that need to search the web, scrape
  pages, extract structured data, interact with dynamic content, crawl entire
  websites, or monitor pages for changes. Agents should reach for Firecrawl when
  they need clean, LLM-ready web data in markdown or JSON format.
metadata:
  mintlify-proj: firecrawl
  version: "1.0"

Firecrawl Skill

Product Summary

Firecrawl is the context API for AI agents. It provides a unified interface to search the web, scrape any page, extract structured data, interact with dynamic content, crawl entire websites, and monitor pages for changes. All endpoints return clean Markdown or structured JSON optimized for LLM consumption.

Key files and commands:

• Base URL: https://api.firecrawl.dev/v2
• Authentication: Authorization: Bearer fc-YOUR-API-KEY (or keyless for /scrape, /search, /parse)
• SDKs: Python (firecrawl), Node.js (firecrawl), CLI (firecrawl-cli), Go, Ruby, Rust, PHP, Elixir, Java, .NET
• Primary docs: https://docs.firecrawl.dev

Core endpoints:

• POST /scrape — Extract content from a single URL
• POST /batch/scrape — Scrape multiple URLs in parallel
• POST /crawl — Recursively crawl an entire website
• POST /search — Search the web and scrape results
• POST /interact — Click, fill forms, navigate pages
• POST /monitor — Schedule recurring checks and detect changes
• POST /agent — Autonomous web data gathering with AI
• POST /parse — Parse local files (PDFs, DOCX, HTML, etc.)

When to Use

Reach for Firecrawl when:

• Scraping a single page: Extract markdown, HTML, JSON, screenshots, or structured data from a URL
• Scraping multiple URLs: Use batch scrape to process a list of URLs concurrently
• Crawling entire sites: Recursively discover and scrape all pages on a domain
• Searching and scraping: Search the web and get full page content from results
• Extracting structured data: Use JSON mode with a schema or prompt to extract specific fields
• Interacting with dynamic content: Click buttons, fill forms, navigate pages, then scrape
• Monitoring for changes: Schedule recurring checks on pages or entire sites and get alerts
• Autonomous research: Use the Agent endpoint when you don't know URLs upfront and need deep web navigation
• Parsing local files: Convert PDFs, DOCX, images, HTML to markdown or structured data
• Handling JavaScript-heavy sites: Firecrawl renders JavaScript and handles anti-bot measures automatically

Quick Reference

Essential Scrape Parameters

Parameter	Type	Default	Purpose
url	string	required	URL to scrape
formats	array	["markdown"]	Output formats: markdown, html, json, screenshot, links, images, product, branding, audio, video, question, highlights
maxAge	integer	172800000	Cache freshness in ms (2 days); set to 0 for fresh content
proxy	string	"auto"	Proxy type: basic, enhanced, or auto
mobile	boolean	false	Emulate mobile device
waitFor	integer	0	Delay in ms before fetching (for JS rendering)
headers	object	—	Custom headers (cookies, user-agent, etc.)
location	object	{country: "US"}	ISO country code and preferred languages

JSON Extraction (Structured Data)

# With schema
result = firecrawl.scrape(
    url="https://example.com",
    formats=[{
        "type": "json",
        "schema": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "price": {"type": "number"}
            }
        }
    }]
)

# Without schema (LLM chooses structure)
result = firecrawl.scrape(
    url="https://example.com",
    formats=[{
        "type": "json",
        "prompt": "Extract product name and price"
    }]
)

Batch Scrape Multiple URLs

result = firecrawl.batch_scrape(
    urls=["https://example.com/page1", "https://example.com/page2"],
    formats=["markdown"],
    max_concurrency=5
)

Crawl Entire Website

result = firecrawl.crawl(
    url="https://example.com",
    limit=100,
    scrape_options={
        "formats": ["markdown"],
        "only_main_content": True
    }
)

Search and Scrape

result = firecrawl.search(
    query="AI agents",
    scrape_options={
        "formats": ["markdown"]
    }
)

Interact with Dynamic Content

# Create browser session
session = firecrawl.create_browser_session()

# Execute actions
result = firecrawl.interact(
    session_id=session.id,
    actions=[
        {"type": "click", "selector": "#search-button"},
        {"type": "write", "text": "firecrawl"},
        {"type": "wait", "milliseconds": 2000},
        {"type": "scrape"}
    ]
)

Monitor Page for Changes

monitor = firecrawl.create_monitor(
    name="pricing-monitor",
    targets=[{
        "type": "scrape",
        "urls": ["https://example.com/pricing"]
    }],
    schedule={"text": "every 6 hours"},
    webhook={"url": "https://your-webhook.com"}
)

Agent (Autonomous Research)

result = firecrawl.agent(
    prompt="Find the latest AI research papers on arxiv.org and extract title, authors, and abstract"
)

Decision Guidance

Scenario	Use This	Why
Single known URL, need markdown	/scrape with formats: ["markdown"]	Synchronous, 1 credit, fast
Single known URL, need structured data	/scrape with formats: [{type: "json", schema: {...}}]	Deterministic extraction, 5 credits (1 base + 4 LLM)
Multiple URLs, same options	/batch/scrape	Parallel processing, efficient
Entire website, unknown URLs	/crawl	Discovers via sitemap + link traversal
Search + scrape results	/search with scrapeOptions	One call for search + full content
Dynamic content (clicks, forms)	/interact	Browser session, stateful
Don't know URLs, need research	/agent	Autonomous navigation, deep search
Local file (PDF, DOCX)	/parse	For files not accessible by URL
Monitor for changes	/monitor	Scheduled checks, webhooks, alerts
Product pages	/scrape with formats: ["product"]	Deterministic, no schema needed, cheaper
JavaScript-heavy site	/scrape with waitFor or /interact	Firecrawl renders JS automatically

Workflow

Typical Scraping Task

1. Understand the requirement: What data do you need? Single page or multiple? Structured or markdown?
2. Check if URL is accessible: Verify the URL is public and not behind authentication
3. Choose the right endpoint:
   • Single URL → /scrape
   • Multiple URLs → /batch/scrape
   • Entire site → /crawl
   • Search first → /search
   • Dynamic content → /interact
   • Unknown URLs → /agent
4. Set output format:
   • Markdown for LLM context: formats: ["markdown"]
   • Structured data: formats: [{type: "json", schema: {...}}]
   • Product data: formats: ["product"]
5. Configure options:
   • Cache: maxAge: 0 for fresh, maxAge: 600000 for 10-min cache
   • Proxy: proxy: "auto" for anti-bot sites
   • Mobile: mobile: true for responsive testing
   • Location: location: {country: "DE"} for geo-specific content
6. Handle response:
   • Check success: true and data.metadata.statusCode (2xx or 304 = clean load)
   • Extract data from data.markdown, data.json, data.product, etc.
   • Handle errors with retry logic (see Common Gotchas)
7. Verify results: Spot-check extracted data matches expectations

Batch Processing Workflow

1. Prepare list of URLs
2. Call /batch/scrape with urls array and scrape_options
3. Get back job ID
4. Poll GET /batch/scrape/{id} until status: "completed"
5. Paginate through results if response exceeds 10MB (use next parameter)
6. Process each page in data array

Crawl Workflow

1. Start crawl with POST /crawl (returns job ID immediately)
2. Poll GET /crawl/{id} to check status
3. When status: "completed", retrieve all pages from data array
4. For large crawls, paginate using next parameter
5. Check GET /crawl/{id}/errors for failed pages (network errors, robots.txt blocks)

Common Gotchas

• Cache vs. freshness: By default, Firecrawl returns cached results (2-day window). Set maxAge: 0 to force fresh scrape, but this is slower and more likely to fail. Use maxAge: 600000 for a 10-minute cache window as a middle ground.

• Status code confusion: The HTTP response status (200) is different from the page's status code (data.metadata.statusCode). Always check both: first confirm success: true, then check statusCode for the actual page outcome.

• Batch scrape vs. crawl: Batch scrape is for a known list of URLs; crawl discovers URLs automatically. Don't use batch scrape for entire sites — use crawl instead.

• JSON extraction costs: JSON mode adds 4 credits per page. For simple markdown, use formats: ["markdown"] (1 credit). For structured data, use formats: ["json"] only when needed.

• Product format is deterministic: The product format extracts from structured data (JSON-LD, schema.org, etc.) without an LLM call. It's cheaper and faster than JSON mode for product pages, but fails silently on non-product pages.

• Crawl scope by default: Crawl only follows child paths by default. Use crawlEntireDomain: true to include siblings/parents, allowSubdomains: true for subdomains, allowExternalLinks: true to follow off-domain links.

• Regex patterns in crawl: includePaths and excludePaths match URL pathname (not full URL) by default. Use regexOnFullURL: true to match query parameters. Patterns are Rust regex (RE2-style); look-around and backreferences not supported.

• Prompt injection risk: When scraping untrusted URLs with LLM-backed formats (json, question, highlights), set checkPromptInjection: true to scan for injection attempts before extraction. Adds 4 credits if check runs.

• PII redaction: Set redactPII: true to redact personally identifiable information from markdown. Adds 4 credits. Choose mode: accurate (model-only, precise), aggressive (heuristics + model, higher recall), or fast (heuristics only, no model call).

• Interact vs. actions: The newer /interact endpoint is stateful and more powerful than the actions parameter on /scrape. Use /interact for complex multi-step workflows; use actions only for simple pre-scrape interactions.

• Webhook signature verification: Always verify the X-Firecrawl-Signature header (HMAC-SHA256) on incoming webhooks. Never process unverified webhooks.

• Rate limits: 429 responses mean you've hit rate limits. Implement exponential backoff. SDKs retry automatically for transient errors (408, 409, 5xx) but not for 429.

• Result expiration: Crawl and batch scrape results are available for 24 hours via API. After that, view them in activity logs only.

• Lockdown mode: Set lockdown: true to serve only from cache (no outbound requests). Useful for compliance/air-gapped environments. Costs 5 credits on hit, 1 on miss.

• Zero data retention: Set zeroDataRetention: true to prevent caching/logging. Some options force this automatically (e.g., actions, headers, profile).

Verification Checklist

Before submitting work with Firecrawl:

• Confirmed success: true in response
• Checked data.metadata.statusCode is 2xx or 304 (not 4xx/5xx)
• Verified extracted data matches expected format (markdown, JSON, product, etc.)
• Spot-checked 2–3 samples of extracted content for accuracy
• Confirmed credit usage is within budget (1 base + format surcharges)
• For batch/crawl jobs: confirmed status: "completed" before processing results
• For webhooks: verified X-Firecrawl-Signature header
• For sensitive data: confirmed storeInCache: false or zeroDataRetention: true if needed
• For dynamic content: confirmed page loaded fully (check waitFor or /interact actions)
• For crawls: checked GET /crawl/{id}/errors for failed pages

Resources

Comprehensive navigation:

• llms.txt — Full index of all documentation pages for agent navigation

Critical documentation:

1. Introduction & Quick Start — Overview of Search, Scrape, Interact
2. Scrape Feature Docs — All scrape formats, extraction, caching, actions
3. API Reference v2 — Complete endpoint reference with examples
4. Crawl Feature Docs — Crawl configuration, scope, webhooks, error handling
5. Agent Feature Docs — Autonomous research without URLs
6. Error Reference — Error codes, causes, retry guidance
7. Rate Limits — Plan limits and concurrency
8. SDKs Overview — Python, Node, CLI, and community SDKs

---

For additional documentation and navigation, see: https://docs.firecrawl.dev/llms.txt