Set up AssemblyAI with an AI agent

name: assemblyai
description: "Use when building voice AI applications: transcribing pre-recorded
  audio or live streams, synthesizing speech, extracting insights from audio, or
  applying guardrails for compliance. Agents should reach for this skill when
  implementing speech-to-text, text-to-speech, speech understanding, or voice
  agent features."
metadata:
  mintlify-proj: assemblyai
  version: "1.0"

AssemblyAI Skill

Product summary

AssemblyAI is a Voice AI infrastructure platform providing speech-to-text (pre-recorded and streaming), text-to-speech, speech understanding, guardrails, and LLM access. Agents use it to transcribe audio, synthesize speech, extract insights (summarization, sentiment, entities), and apply compliance controls. The platform offers official SDKs for Python and JavaScript, raw HTTP/WebSocket APIs, and integrations with voice agent frameworks (LiveKit, Pipecat, Vapi, Bolna) and no-code tools (Zapier, Make, n8n).

Key files and commands:

• Environment variable: ASSEMBLYAI_API_KEY (set once, used by all SDKs)
• Python SDK: pip install assemblyai
• JavaScript SDK: npm install assemblyai
• Primary docs: https://assemblyai.com/docs
• Comprehensive page listing: https://assemblyai.com/docs/llms.txt

When to use

Reach for this skill when:

• Transcribing pre-recorded audio: audio files, video, podcasts, call recordings (use Pre-recorded STT API)
• Real-time transcription: live microphone input, streaming audio, voice agents (use Streaming STT API)
• Short synchronous transcription: clips up to 20 minutes, no polling needed (use Sync STT API)
• Text-to-speech: generating voice for agents, replies, or playback (use TTS API)
• Extracting audio insights: summarization, sentiment, entities, topics, speaker identification (use Speech Understanding API)
• Compliance and safety: PII redaction, content moderation, profanity filtering (use Guardrails API)
• Unified LLM access: applying reasoning models to transcripts without separate integrations (use LLM Gateway API)
• Building voice agents: combining STT, TTS, and LLM reasoning in frameworks like LiveKit or Pipecat

Quick reference

Model selection

Use case	Pre-recorded	Streaming	Notes
Highest accuracy, 18 languages	Universal-3.5 Pro	Universal-3.6 Pro Realtime	Recommended default
99 languages, cost-effective	Universal-2	Universal-Streaming Multilingual	Fallback for unsupported languages
English only, lowest cost	—	Universal-Streaming English	Fastest and cheapest for EN
Medical terminology	Medical Mode (add-on)	Medical Mode (add-on)	Works with any speech model

API endpoints

Product	Protocol	Endpoint
Pre-recorded STT	HTTP	POST https://api.assemblyai.com/v2/transcript
Streaming STT	WebSocket	wss://streaming.assemblyai.com/v3/ws
Sync STT	HTTP	POST https://sync.assemblyai.com/v1/transcribe
Text-to-Speech	WebSocket/HTTP	wss://tts.assemblyai.com/v1 or POST https://tts.assemblyai.com/v1/synthesize
Speech Understanding	HTTP	POST https://llm-gateway.assemblyai.com/v1/understanding
LLM Gateway	HTTP	POST https://llm-gateway.assemblyai.com/v1/chat/completions

Common transcription options

# Pre-recorded (Python SDK)
config = TranscriptionConfig(
    speaker_labels=True,           # Split by speaker
    language_detection=True,       # Auto-detect language
    speech_models=["universal-3-5-pro"],  # Model selection
    entity_detection=True,         # Extract entities
    sentiment_analysis=True,       # Sentiment per utterance
    auto_chapters=True,            # Auto-generate chapters
    summarization=True,            # Generate summary
    webhook_url="https://...",     # Async notification
)
transcript = transcriber.transcribe(audio_url, config=config)

Rate limits

Product	Free	Paid	Notes
Pre-recorded STT	5 parallel	200+ parallel	Queued beyond limit
Streaming STT	5 new sessions/min	200 new sessions/min	Auto-scales with usage
Speech Understanding	2 requests/min	30 requests/min	Per account, 60-sec window
HTTP requests	20,000/5 min	20,000/5 min	Counted across all endpoints

Decision guidance

When to use X vs Y

Scenario	Use	Why
Audio already recorded, up to 10 hours	Pre-recorded STT	Async, no polling overhead, highest accuracy
Live microphone or streaming source	Streaming STT	Real-time results, ~150ms p50 latency, billed per session duration
Short clip (≤20 min), need immediate response	Sync STT	Single HTTP request-response, no polling, no webhooks
Dictation with formatting	Dictation API	LLM formats text in same request as transcription
Multiple speakers, need labels	Enable speaker_labels=True	Diarization splits utterances by speaker
Automatic language detection	Set language_detection=True	Detects dominant language; can misidentify accented speech
Manual language override	Set language_code="es"	Use when auto-detection fails or language is known
Polling for status	GET /v2/transcript/{id}	Limit 60 requests/min per transcript; use webhooks for high volume
Webhooks for completion	Set webhook_url on submit	Better for background jobs; avoids polling rate limits
Streaming session cost control	Close session when done	Billed per session duration (time connection open), not audio sent
Voice agent with TTS	Use TTS WebSocket session	Keep one session open per conversation; send replies as they arrive

Workflow

Transcribe pre-recorded audio

1. Set API key: Export ASSEMBLYAI_API_KEY environment variable once.
2. Choose model: Default is ["universal-3-5-pro", "universal-2"] (auto-fallback for unsupported languages). Override with speech_models parameter if needed.
3. Prepare audio: Use public URL or upload local file (SDK handles upload automatically).
4. Submit transcription: Call transcriber.transcribe(audio_url, config=config) (SDK) or POST /v2/transcript (HTTP).
5. Wait for completion: SDK polls automatically; HTTP requires polling GET /v2/transcript/{id} every 3 seconds or use webhooks.
6. Check status: Transcript has status processing, queued, completed, or error. Log transcript.id for every request (required for support).
7. Extract results: Access transcript.text (full text), transcript.utterances (per-speaker), transcript.entities (if enabled), etc.
8. Handle errors: Check transcript.status == "error" and transcript.error message; include transcript ID in support requests.

Stream live audio

1. Set API key: Export ASSEMBLYAI_API_KEY environment variable.
2. Choose model: Default is universal-3-6-pro (recommended). Use universal-3-5-pro or universal-streaming-multilingual for cost savings.
3. Create transcriber: Instantiate RealTimeTranscriber (Python) or streaming.transcriber() (JavaScript).
4. Connect: Call client.connect(RealTimeParameters(...)) to open WebSocket.
5. Stream audio: Send audio chunks via client.stream(chunk) as they arrive.
6. Handle events: Listen for Turn events (partial and final transcripts), Begin (session opened), Termination (session closed).
7. Close session: Call client.disconnect(terminate=True) when done; finalizes open turn and closes connection.
8. Billing note: Charged for entire session duration (time connection open), not audio sent. Always close sessions.

Synthesize speech with TTS

1. Set API key: Export ASSEMBLYAI_API_KEY environment variable.
2. Create session: Call client.tts.session(voiceId="aai-kyla", language="english").
3. Connect: Call session.connect() to open WebSocket and load voice.
4. Send text: Call session.speak(text) to synthesize; returns async iterator of audio chunks.
5. Play audio: Write chunks to file or stream to playback device.
6. Keep alive: Call session.keepAlive() during idle periods to prevent timeout.
7. Close: Call session.close() when conversation ends.

Extract insights from transcript

1. Transcribe first: Get transcript ID from pre-recorded or streaming transcription.
2. Choose features: Enable on transcription request (entity_detection=True, sentiment_analysis=True, etc.) or call Speech Understanding API post-transcription.
3. Access results: Utterances include sentiment, entities, speaker (if diarization enabled), etc.
4. Use LLM Gateway: For complex analysis, send transcript to LLM Gateway with POST /v1/chat/completions for reasoning.

Common gotchas

• API key in client-side code: Never ship API key to browser. Use temporary tokens (POST /v1/streaming/token) for client-side streaming.
• File upload format: Use curl --data-binary @file (raw bytes), not -d or JSON body, or transcoding fails with "File type application/json" error.
• Polling rate limit: 60 GET requests per minute per transcript. Use webhooks for high-volume jobs or jittered polling.
• Streaming session billing: Charged for entire session duration (time connection open), not audio sent. Idle time counts. Always send Terminate message and close connection.
• Language detection misidentification: Accented speech can be detected as wrong language (e.g., accented English detected as Finnish). Set language_code manually if auto-detection fails.
• Speaker labels require diarization: Enable speaker_labels=True to split utterances by speaker; without it, you get one continuous transcript.
• Transcript ID required for support: Always log transcript.id with timestamp and region; support needs it to look up requests.
• Webhook timeout: Endpoint must return 2xx within 10 seconds or AssemblyAI retries up to 10 times. 4xx stops retries.
• Model fallback: If language not supported by chosen model, system falls back to next in speech_models array. Check speech_model_used in response.
• Temporary token expiration: Temporary tokens expire; regenerate before session starts if reusing old token.

Verification checklist

Before submitting work:

• API key set as environment variable (ASSEMBLYAI_API_KEY), not hardcoded
• Transcript ID logged for every request (required for debugging)
• Error handling checks transcript.status == "error" and includes error message
• Streaming sessions closed with terminate=True to finalize open turn
• Webhooks return 2xx status within 10 seconds (or polling used instead)
• File uploads use --data-binary (raw bytes), not JSON or file-path string
• Language code set manually if auto-detection unreliable (accented speech)
• Speaker labels enabled if per-speaker utterances needed
• Rate limits respected: 60 polling requests/min per transcript, 20k HTTP/5min total
• Model selection matches use case (U3.5 Pro for accuracy, U2 for 99 languages, streaming model for real-time)
• Temporary tokens regenerated before expiration if using client-side streaming

Resources

• Comprehensive page listing: https://assemblyai.com/docs/llms.txt
• Pre-recorded STT quickstart: https://assemblyai.com/docs/pre-recorded-audio/getting-started/transcribe-an-audio-file
• Streaming STT quickstart: https://assemblyai.com/docs/streaming/getting-started/transcribe-streaming-audio
• Models and pricing: https://assemblyai.com/docs/getting-started/models

---

For additional documentation and navigation, see: https://assemblyai.com/docs/llms.txt