Set up Tavily with an AI agent

name: tavilyai
description: Use when building agents that need web search, content extraction,
  site crawling, or research capabilities. Reach for Tavily when your agent must
  access current information, pull content from URLs, map site structures, or
  synthesize multi-source research reports.
metadata:
  mintlify-proj: tavilyai
  version: "1.0"

Tavily Skill

Product Summary

Tavily is an AI-optimized web layer for agents: real-time search, page extraction, site crawling and mapping, and cited research. It returns clean, LLM-ready content instead of raw HTML or shallow snippets, reducing token waste and improving agent reliability.

Key files and commands:

• CLI: tvly (install via curl -fsSL https://cli.tavily.com/install.sh | bash)
• SDKs: Python (tavily-python), JavaScript (@tavily/core)
• MCP server: https://mcp.tavily.com/mcp/ (for Claude, Cursor, ChatGPT, Codex, etc.)
• API endpoints: /search, /extract, /crawl, /map, /research
• Free tier: 1,000 credits/month (no credit card required)
• Keyless access: /search and /extract work with zero setup via X-Tavily-Access-Mode: keyless header

Primary docs: https://docs.tavily.com

When to Use

Reach for Tavily when:

• Agent needs current information — search the web for real-time data, news, or recent developments
• Extract content from URLs — pull clean markdown/text from 1–20 URLs without parsing HTML or handling JavaScript
• Crawl a site — traverse multiple pages and extract content from a domain (e.g., documentation, competitive research)
• Map site structure — discover URLs on a site without extracting content (faster than crawling)
• Research a topic — synthesize multi-source research into a cited report
• Enrich data — fill in company/person records from the web (via Agent Skills)
• Monitor for changes — track news, pricing pages, or job postings on a cadence

Do not use Tavily for:

• Accessing private/authenticated content (Tavily only crawls public web)
• Bypassing robots.txt or site terms of service
• Bulk scraping without respecting rate limits and site policies

Quick Reference

API Endpoints & Credits

Endpoint	Cost	Use case
/search	1 credit (basic/fast/ultra-fast), 2 credits (advanced)	Web search with ranked results
/extract	1 credit per 5 URLs (basic), 2 credits per 5 URLs (advanced)	Pull content from URLs
/crawl	1 credit per 10 pages (basic), 2 credits per 10 pages (advanced)	Traverse and extract from a site
/map	1 credit per 10 URLs	Discover site structure
/research	Variable (multi-step)	Comprehensive research report

CLI Commands

tvly search "query"                    # Search the web
tvly extract <url> [<url> ...]        # Extract from 1–20 URLs
tvly crawl <url>                      # Crawl a site
tvly map <url>                        # Discover URLs
tvly research "topic"                 # Deep research
tvly login --api-key tvly-YOUR_KEY    # Authenticate
tvly --status                         # Check auth status

Authentication

Method	When to use
Keyless (X-Tavily-Access-Mode: keyless header)	Testing, light use, /search and /extract only
API key (Authorization: Bearer tvly-YOUR_KEY)	Production, all endpoints, higher limits
OAuth (MCP clients)	Claude, Cursor, ChatGPT — sign in via browser
Environment variable (TAVILY_API_KEY)	CLI and SDKs

Search Depth Trade-offs

Depth	Latency	Relevance	Content type	Cost	Best for
ultra-fast	Lowest	Lower	Summary	1 credit	Real-time, speed-critical
fast	Low	Good	Chunks	1 credit	Quick, targeted snippets
basic	Medium	High	Chunks	1 credit	General-purpose (default)
advanced	Higher	Highest	Chunks	2 credits	Niche topics, detailed queries

Extract Parameters

Parameter	Use case
query	Rerank chunks by relevance to a query (enables chunks_per_source)
chunks_per_source	Limit content per URL (1–5 chunks, ~800 chars each)
extract_depth: advanced	Handle JavaScript-rendered pages, tables, media
format: markdown or text	Output format

Crawl Parameters

Parameter	Use case
max_depth	How many levels deep to crawl (1–5)
max_breadth	Links to follow per page (default 20)
limit	Total page cap (default 50)
instructions	Natural language guidance (e.g., "only API docs")
select_paths / exclude_paths	Regex patterns to filter URLs
chunks_per_source	Relevant chunks per page (requires instructions)

Decision Guidance

When to use Search vs Extract vs Crawl

Scenario	Use
"Find recent news about X"	/search with topic: news
"Get content from this URL"	/extract (1–20 URLs)
"Get content from multiple URLs I already have"	/extract (batch up to 20)
"Explore a site and extract from many pages"	/crawl (auto-discover and extract)
"Just find URLs on a site, don't extract yet"	/map (faster, cheaper)
"Synthesize research from many sources"	/research (multi-step, cited report)

When to use Search Depth

Query type	Depth	Reason
"Latest AI news"	basic	Fast, general-purpose
"Niche technical topic"	advanced	Reaches more sources, higher relevance
"Real-time stock price"	ultra-fast	Speed matters most
"Detailed comparison of X vs Y"	advanced	Complex query benefits from broader search

When to use Extract Depth

Content	Depth
Simple blog posts, news articles	basic
JavaScript-heavy SPAs, tables, embedded media	advanced

Search with include_raw_content vs Extract API

Approach	When to use
search(..., include_raw_content=True)	Quick prototyping, search results are likely relevant
extract(urls=[...])	You already have specific URLs; need control over which ones

Workflow

Typical agent task: Research a topic

1. Understand the request — What does the user need? A quick fact, a comparison, a report, or ongoing monitoring?

2. Choose the right endpoint:

   • Quick fact → /search with max_results: 3–5
   • Detailed comparison → /search with search_depth: advanced, max_results: 10
   • Comprehensive report → /research (multi-step, cited)
   • Extract from known URLs → /extract

3. Set parameters:

   • For search: set search_depth, max_results, topic (news/general/finance), date filters if needed
   • For extract: set query (to rerank chunks), chunks_per_source (1–5), extract_depth if JavaScript-heavy
   • For crawl: set max_depth, limit, instructions (natural language guidance)

4. Make the request:

   from tavily import TavilyClient
   client = TavilyClient(api_key="tvly-YOUR_KEY")
   response = client.search("your query", search_depth="basic", max_results=5)
   

5. Handle the response:

   • Check response['results'] for search results (title, url, content, score)
   • Check response['answer'] if include_answer was set
   • Check response['failed_results'] for extract/crawl failures per URL

6. Filter and deduplicate:

   • Use score to filter low-relevance results (e.g., score > 0.5)
   • Deduplicate URLs across multiple searches (canonicalize by removing ? params and trailing /)
   • Combine unique chunks with [...] separator to save tokens

7. Verify and return:

   • Confirm results are relevant and current
   • Pass include_usage: True to track credits spent
   • Return results to the user or feed into downstream processing

Typical agent task: Extract content from URLs

1. Gather URLs — From search results, user input, or prior crawl

2. Batch up to 20 URLs — Extract accepts 1–20 per request

3. Set extraction parameters:

   • If you have a query: set query + chunks_per_source (1–5) for focused extraction
   • If you need full content: omit query (returns all content)
   • If page is JavaScript-heavy: set extract_depth: advanced

4. Make the request:

   response = client.extract(
       urls=["https://example.com/page1", "https://example.com/page2"],
       query="authentication API",
       chunks_per_source=3,
       extract_depth="basic"
   )
   

5. Handle results:

   • response['results'] contains successful extractions (url, raw_content)
   • response['failed_results'] lists URLs that failed (url, error)
   • Retry failed URLs individually or skip them

6. Process content:

   • Parse markdown/text from raw_content
   • If query was set, chunks are already reranked by relevance
   • Combine chunks with [...] separator if needed

Typical agent task: Crawl a site

1. Start with /map — Discover structure first, cheaper than crawling

2. Set map parameters:

   tvly map https://docs.example.com --max-depth 2 --limit 100
   

3. Review discovered URLs — Decide if you need to crawl or extract specific pages

4. Crawl if needed:

   tvly crawl https://docs.example.com \
     --max-depth 2 \
     --instructions "focus on API reference pages" \
     --chunks-per-source 3 \
     --output-dir ./docs
   

5. Monitor results:

   • Check response['results'] for extracted pages
   • Check response['failed_results'] for pages that failed
   • Adjust max_depth, limit, or instructions if results are incomplete

Common Gotchas

• Keyless access is rate-limited — Test with keyless, but upgrade to an API key for production. Keyless responses are identical to keyed responses; no code changes needed.

• Search returns chunks, not full pages — By default, /search returns 3 snippets per source (~500 chars each). Use /extract if you need full page content.

• Extract accepts up to 20 URLs per request — Batch them, but don't exceed 20 in one call. Retry failed URLs individually.

• Crawl can be expensive — Start with max_depth: 1, limit: 50. Increase gradually. Use instructions to focus crawling on relevant pages.

• Query parameter is required for chunks_per_source — You can't use chunks_per_source without query in extract/crawl. If you need to limit content without a query, use include_raw_content: false in search.

• robots.txt and rate limits matter — Respect site policies. Implement delays between requests. Monitor API usage with /usage endpoint.

• Session tracking is optional but recommended — Pass session_id and human_id across multi-step workflows so Tavily can link related calls and improve response quality.

• Search depth affects cost — advanced costs 2 credits vs 1 for basic/fast/ultra-fast. Set it explicitly to control costs; don't rely on auto_parameters unless you want the extra cost.

• Extract depth affects latency — advanced extraction is slower but handles JavaScript-rendered pages and tables better. Use basic for simple content.

• Date filters don't guarantee freshness — time_range filters by publish date, but cached pages may be old. Use max_age_hours: 0 to always fetch live (adds latency).

• Exact match narrows results — exact_match: true requires the phrase to appear verbatim. May return fewer results or empty fields.

Verification Checklist

Before submitting work:

• API key is set — Check TAVILY_API_KEY env var or passed to client
• Endpoint is correct — Use /search for discovery, /extract for content, /crawl for multi-page, /research for reports
• Parameters are reasonable — max_results ≤ 20, chunks_per_source 1–5, max_depth 1–5, crawl limit ≤ 100
• Search depth is intentional — basic by default; use advanced only when needed (costs 2 credits)
• Batch size is correct — Extract/crawl: up to 20 URLs per request
• Error handling is in place — Check failed_results, retry with backoff, don't fail on single URL errors
• Rate limits are respected — Cap concurrent requests (e.g., 5–20 in-flight), implement exponential backoff on 429
• Session tracking is used — Pass session_id and human_id for multi-step workflows
• Results are deduplicated — Canonicalize URLs, combine unique chunks
• Credits are tracked — Pass include_usage: True, sum usage across batch
• Content is validated — Confirm results are relevant, current, and not duplicates

Resources

Comprehensive navigation: https://docs.tavily.com/llms.txt (page-by-page index for agents)

Critical docs:

• API Reference — All endpoints, parameters, response schemas
• Best Practices for Search — Query optimization, depth trade-offs, batch workflows, deduplication
• Best Practices for Extract — Extraction strategies, filtering, multi-step pipelines

---

For additional documentation and navigation, see: https://docs.tavily.com/llms.txt