Set up Tavus with an AI agent

name: tavus
description: Use when building real-time conversational video AI agents (PALs),
  creating and training custom faces, deploying conversations to web apps,
  integrating tool calling, or managing multi-turn video interactions with
  perception and speech recognition.
metadata:
  mintlify-proj: tavus
  version: "1.0"

Tavus Skill

Product Summary

Tavus is a platform for building real-time, multimodal conversational video interfaces (CVI). You build a PAL (Personified Application Layer) — a configurable AI agent with behavior, knowledge, tools, and pipeline layers — pair it with a Face (a photorealistic avatar trained from video or image), and deploy it as a live video conversation. The core API endpoint is https://tavusapi.com/v2/. Authenticate with an API key in the x-api-key header. Key resources: API Overview, CVI Quickstart, PAL Maker.

When to Use

Reach for Tavus when:

• Building a real-time video conversation agent (sales rep, interviewer, customer support, medical intake)
• Creating or training a custom face from video or image (Phoenix-4.5 or Phoenix-4 models)
• Deploying a conversation to a website as a widget, embed, or landing page
• Wiring tool calls into a PAL so it can look things up, write to a CRM, or trigger actions
• Managing multi-turn conversations with perception (visual/audio understanding) and speech recognition
• Storing participant memory across sessions (pinned facts and learned context)
• Setting objectives (workflow steps) or guardrails (behavioral boundaries) for structured conversations

Quick Reference

Core API Endpoints

Operation	Endpoint	Use
Create PAL	POST /v2/pals	Define behavior, system prompt, pipeline layers
Create Face	POST /v2/faces	Train from video or image (Phoenix-4.5/4)
Create Conversation	POST /v2/conversations	Start a live video call session
Create Deployment	POST /v2/deployments	Publish PAL to widget/embed/landing-page
Create Tool	POST /v2/tools	Define a function the PAL can call
Attach Tool	POST /v2/pals/{pal_id}/tools	Wire tool onto a PAL
Create Memory Store	POST /v2/memory-stores	Store participant context across calls
Create Objectives	POST /v2/objectives	Define workflow steps and branching
Create Guardrails	POST /v2/guardrails	Set behavioral boundaries

PAL Configuration Fields

Field	Type	Purpose
pal_name	string	Display name
system_prompt	string	Core behavior and tone (keep under 5,000 tokens)
default_face_id	string	Stock face ID or trained face ID
pipeline_mode	string	full (perception + STT + LLM + TTS) or llm_only
layers.perception.perception_model	string	raven-1 for visual/audio understanding
layers.conversational_flow.turn_detection_model	string	sparrow-2 (recommended) for turn-taking
layers.llm.model	string	tavus-gemma (default, fastest), tavus-gpt-sol, tavus-gpt-terra
layers.tts.voice_id	string	Voice for speech synthesis
objectives_id	string	Workflow structure (templated conversations)
guardrails_id	string	Behavioral boundaries

Face Training Models

Model	Input	Speed	Best For
Phoenix-4.5	Image or video	Fast (preview in ~1 min)	Portrait, quick turnaround
Phoenix-4	Video only	Medium (3–4 hours)	Full-body, high quality
Phoenix-3	Video only	Legacy	Existing integrations only

Deployment Channels

Channel	Element	Use
Widget	<tavus-widget deployment-id="...">	Floating launcher on existing site
Embed	<tavus-embed deployment-id="...">	Inline conversation in your layout
Landing page	Hosted by Tavus	Standalone link, no website needed

Tool Types

Type	Trigger	Use
LLM tool	User speech	PAL decides to call based on conversation
Perception tool	Audio/video stream	Raven detects something and calls tool
Post-call action	Conversation end	Run work after call ends (transcript + analysis)
System tool	Always available	Built-in (e.g., end_call)

MCP & CLI

Surface	Best For	Auth
MCP server (https://mcp.tavus.io/mcp)	AI agents building PALs in editor	Browser OAuth or API key
CLI (tavus command)	Humans at terminal, CI automation	TAVUS_API_KEY env var

Decision Guidance

When to Use Full Pipeline vs. LLM-Only

Scenario	Use Full Pipeline	Use LLM-Only
Need perception (see/hear user)	✓
Need speech recognition (STT)	✓
Text-only or external STT		✓
Lowest latency critical		✓
Multimodal interactions	✓

When to Use Objectives vs. Guardrails

Goal	Use Objectives	Use Guardrails
Collect info in defined order	✓
Branch based on answers	✓
Enforce behavioral rules		✓
Flag policy violations		✓
Workflow progression	✓
Safety/compliance steering		✓

When to Use Deployment vs. Direct API

Scenario	Use Deployment	Use Direct API
Drop into website with one tag	✓
No backend needed	✓
Full control over UI		✓
Programmatic conversation creation		✓
Managed call limits/bot protection	✓

When to Use MCP vs. CLI

Scenario	Use MCP	Use CLI
Agent building PAL in editor	✓
Human at terminal		✓
CI/CD automation		✓
Interactive tool wiring	✓

Workflow

Build and Deploy a PAL (API Path)

1. Authenticate: Generate API key in PAL Maker. Send it in x-api-key header on every request.

2. Create a Face (if using custom):

   • For image: POST /v2/faces with model_name: "phoenix-4.5", train_image_url, and voice_id
   • For video: POST /v2/faces with model_name: "phoenix-4.5", train_video_url, and voice_id
   • Or use a stock face ID (e.g., rc9cff32ceba for Anna)

3. Create a PAL: POST /v2/pals with:

   • pal_name, system_prompt (clear, under 5k tokens)
   • default_face_id (stock or trained)
   • pipeline_mode: "full" (for perception + STT)
   • layers with perception model, turn-taking model, LLM model, TTS voice
   • Optional: objectives_id, guardrails_id, documents (knowledge base)

4. Attach Tools (if needed):

   • POST /v2/tools to define each tool (name, description, parameters, delivery)
   • POST /v2/pals/{pal_id}/tools to attach to PAL

5. Create a Conversation: POST /v2/conversations with:

   • pal_id
   • conversational_context (optional: participant info, session data)
   • participant_tags (optional: for memory store)
   • Response includes conversation_url to join

6. Join the Conversation: Click conversation_url or embed it in your app.

Deploy to Website (Deployment Path)

1. Create a PAL (steps 1–4 above)
2. Create a Deployment: POST /v2/deployments with:
   • pal_id
   • channel: "widget" or "embed" or "landing-page"
   • allowed_origins (restrict to your domains)
   • limits (daily/total call caps)
3. Add to HTML:

   <tavus-embed deployment-id="YOUR_DEPLOYMENT_ID"></tavus-embed>
   <script src="https://unpkg.com/@tavus/embed"></script>
   

4. No backend needed: Deployment ID is safe to expose; Tavus manages conversations server-side.

Wire Tool Calls (API Delivery)

1. Create Tool: POST /v2/tools with:

   • origin: "llm" (LLM calls) or "vision" (perception calls)
   • delivery.type: "api"
   • delivery.api.url (your endpoint)
   • delivery.api.auth.type (none, bearer, api_key, basic, oauth2_client_credentials, hmac)

2. Attach to PAL: POST /v2/pals/{pal_id}/tools

3. Handle Requests: Your endpoint receives POST with tool parameters. Return 2xx with result JSON.

4. Test: Start a conversation and watch tool fire. Check response handling.

Build with MCP (Agent-Driven)

1. Connect MCP client to https://mcp.tavus.io/mcp with API key in Authorization: Bearer <key> header.
2. Agent calls tavus_pal_create with PAL config.
3. Agent calls tavus_tool_create to define tools.
4. Agent calls tavus_pal_tools_attach to wire them.
5. Agent calls tavus_conversation_create to test.
6. Agent scaffolds handlers in your codebase for each tool.

Common Gotchas

• System prompt too long: Keep under 5,000 tokens (~20k characters). Longer prompts degrade speed and instruction-following.
• Forgetting x-api-key header: Every API request requires it. Requests without it return 401.
• Reusing face IDs incorrectly: Stock face IDs (e.g., rc9cff32ceba) are fixed; trained faces have unique IDs. Don't mix them up.
• Tool parameters mismatch: If tool definition doesn't match your handler's request shape, the PAL won't call it correctly. Test with tavus_conversation_create first.
• Memory store not persisting: Use stable participant_tags (e.g., user ID) across calls. Changing the tag creates a new store. Learned memory processes asynchronously after calls end; wait a few minutes before starting the next conversation.
• Objectives conflicting with system prompt: If system prompt says "be a tutor" but objectives say "qualify leads," the PAL will be confused. Align them.
• Deployment ID exposed: It's safe—it's meant to be public. Protect it with allowed_origins, call limits, and bot protection (Cloudflare Turnstile).
• Face training rejected: Check image/video requirements (full face visible, facing camera, no accessories, clear speech). Phoenix-4.5 preview has a watermark; tuning removes it.
• Latency issues: Use tavus-gemma (fastest LLM), sparrow-2 (turn-taking), and full pipeline. Avoid oversized prompts. Test with Daily's network check.
• Legacy inline tools: Don't define tools in layers.llm.tools anymore. Create them at /v2/tools and attach them. Inline tools are deprecated.

Verification Checklist

Before submitting a PAL or deployment:

• API key is valid and in x-api-key header
• PAL system_prompt is under 5,000 tokens and clear
• default_face_id is a valid stock or trained face ID
• pipeline_mode matches your use case (full for perception, llm_only for text-only)
• LLM model is one of: tavus-gemma, tavus-gpt-sol, tavus-gpt-terra
• Turn-taking model is sparrow-2 (recommended)
• All tools have origin (llm/vision) and delivery configured
• Tool parameters JSON Schema matches your handler's request shape
• Objectives (if used) don't conflict with system prompt
• Guardrails are clear, specific, and testable
• Deployment has allowed_origins set to your domains
• Deployment has call limits configured
• Conversation created successfully and conversation_url is accessible
• Test a full conversation: user speaks, PAL responds, tools fire (if any)
• Memory store persists context across calls (if using participant_tags)

Resources

• Full docs index: https://docs.tavus.io/llms.txt — comprehensive page-by-page navigation for agents
• API Reference: https://docs.tavus.io/api-reference/overview — all endpoints and schemas
• CVI Quickstart: https://docs.tavus.io/sections/conversational-video-interface/quickstart/cvi-quickstart — create PAL and conversation in minutes
• PAL Overview: https://docs.tavus.io/sections/conversational-video-interface/pal/overview — all PAL fields and layers
• Deployments: https://docs.tavus.io/sections/deployments/overview — widget, embed, landing page
• MCP Server: https://docs.tavus.io/sections/agent-tools/mcp-server — agent-driven PAL building
• Prompting Guide: https://docs.tavus.io/sections/onboarding-guide/prompting-guide — system prompt best practices

---

For additional documentation and navigation, see: https://docs.tavus.io/llms.txt