---
name: assemblyai
description: "Use when building voice AI applications: transcribing pre-recorded
  audio or live streams, synthesizing speech, extracting insights from audio, or
  applying guardrails for compliance. Agents should reach for this skill when
  implementing speech-to-text, text-to-speech, speech understanding, or voice
  agent features."
metadata:
  mintlify-proj: assemblyai
  version: "1.0"
---

# AssemblyAI Skill

## Product summary

AssemblyAI is a Voice AI infrastructure platform providing speech-to-text (pre-recorded and streaming), text-to-speech, speech understanding, guardrails, and LLM access. Agents use it to transcribe audio, synthesize speech, extract insights (summarization, sentiment, entities), and apply compliance controls. The platform offers official SDKs for Python and JavaScript, raw HTTP/WebSocket APIs, and integrations with voice agent frameworks (LiveKit, Pipecat, Vapi, Bolna) and no-code tools (Zapier, Make, n8n).

**Key files and commands:**
- Environment variable: `ASSEMBLYAI_API_KEY` (set once, used by all SDKs)
- Python SDK: `pip install assemblyai`
- JavaScript SDK: `npm install assemblyai`
- Primary docs: https://assemblyai.com/docs
- Comprehensive page listing: https://assemblyai.com/docs/llms.txt

## When to use

Reach for this skill when:
- **Transcribing pre-recorded audio**: audio files, video, podcasts, call recordings (use Pre-recorded STT API)
- **Real-time transcription**: live microphone input, streaming audio, voice agents (use Streaming STT API)
- **Short synchronous transcription**: clips up to 20 minutes, no polling needed (use Sync STT API)
- **Text-to-speech**: generating voice for agents, replies, or playback (use TTS API)
- **Extracting audio insights**: summarization, sentiment, entities, topics, speaker identification (use Speech Understanding API)
- **Compliance and safety**: PII redaction, content moderation, profanity filtering (use Guardrails API)
- **Unified LLM access**: applying reasoning models to transcripts without separate integrations (use LLM Gateway API)
- **Building voice agents**: combining STT, TTS, and LLM reasoning in frameworks like LiveKit or Pipecat

## Quick reference

### Model selection

| Use case | Pre-recorded | Streaming | Notes |
|----------|--------------|-----------|-------|
| Highest accuracy, 18 languages | Universal-3.5 Pro | Universal-3.6 Pro Realtime | Recommended default |
| 99 languages, cost-effective | Universal-2 | Universal-Streaming Multilingual | Fallback for unsupported languages |
| English only, lowest cost | — | Universal-Streaming English | Fastest and cheapest for EN |
| Medical terminology | Medical Mode (add-on) | Medical Mode (add-on) | Works with any speech model |

### API endpoints

| Product | Protocol | Endpoint |
|---------|----------|----------|
| Pre-recorded STT | HTTP | `POST https://api.assemblyai.com/v2/transcript` |
| Streaming STT | WebSocket | `wss://streaming.assemblyai.com/v3/ws` |
| Sync STT | HTTP | `POST https://sync.assemblyai.com/v1/transcribe` |
| Text-to-Speech | WebSocket/HTTP | `wss://tts.assemblyai.com/v1` or `POST https://tts.assemblyai.com/v1/synthesize` |
| Speech Understanding | HTTP | `POST https://llm-gateway.assemblyai.com/v1/understanding` |
| LLM Gateway | HTTP | `POST https://llm-gateway.assemblyai.com/v1/chat/completions` |

### Common transcription options

```python
# Pre-recorded (Python SDK)
config = TranscriptionConfig(
    speaker_labels=True,           # Split by speaker
    language_detection=True,       # Auto-detect language
    speech_models=["universal-3-5-pro"],  # Model selection
    entity_detection=True,         # Extract entities
    sentiment_analysis=True,       # Sentiment per utterance
    auto_chapters=True,            # Auto-generate chapters
    summarization=True,            # Generate summary
    webhook_url="https://...",     # Async notification
)
transcript = transcriber.transcribe(audio_url, config=config)
```

### Rate limits

| Product | Free | Paid | Notes |
|---------|------|------|-------|
| Pre-recorded STT | 5 parallel | 200+ parallel | Queued beyond limit |
| Streaming STT | 5 new sessions/min | 200 new sessions/min | Auto-scales with usage |
| Speech Understanding | 2 requests/min | 30 requests/min | Per account, 60-sec window |
| HTTP requests | 20,000/5 min | 20,000/5 min | Counted across all endpoints |

## Decision guidance

### When to use X vs Y

| Scenario | Use | Why |
|----------|-----|-----|
| Audio already recorded, up to 10 hours | Pre-recorded STT | Async, no polling overhead, highest accuracy |
| Live microphone or streaming source | Streaming STT | Real-time results, ~150ms p50 latency, billed per session duration |
| Short clip (≤20 min), need immediate response | Sync STT | Single HTTP request-response, no polling, no webhooks |
| Dictation with formatting | Dictation API | LLM formats text in same request as transcription |
| Multiple speakers, need labels | Enable `speaker_labels=True` | Diarization splits utterances by speaker |
| Automatic language detection | Set `language_detection=True` | Detects dominant language; can misidentify accented speech |
| Manual language override | Set `language_code="es"` | Use when auto-detection fails or language is known |
| Polling for status | GET `/v2/transcript/{id}` | Limit 60 requests/min per transcript; use webhooks for high volume |
| Webhooks for completion | Set `webhook_url` on submit | Better for background jobs; avoids polling rate limits |
| Streaming session cost control | Close session when done | Billed per session duration (time connection open), not audio sent |
| Voice agent with TTS | Use TTS WebSocket session | Keep one session open per conversation; send replies as they arrive |

## Workflow

### Transcribe pre-recorded audio

1. **Set API key**: Export `ASSEMBLYAI_API_KEY` environment variable once.
2. **Choose model**: Default is `["universal-3-5-pro", "universal-2"]` (auto-fallback for unsupported languages). Override with `speech_models` parameter if needed.
3. **Prepare audio**: Use public URL or upload local file (SDK handles upload automatically).
4. **Submit transcription**: Call `transcriber.transcribe(audio_url, config=config)` (SDK) or `POST /v2/transcript` (HTTP).
5. **Wait for completion**: SDK polls automatically; HTTP requires polling `GET /v2/transcript/{id}` every 3 seconds or use webhooks.
6. **Check status**: Transcript has status `processing`, `queued`, `completed`, or `error`. Log `transcript.id` for every request (required for support).
7. **Extract results**: Access `transcript.text` (full text), `transcript.utterances` (per-speaker), `transcript.entities` (if enabled), etc.
8. **Handle errors**: Check `transcript.status == "error"` and `transcript.error` message; include transcript ID in support requests.

### Stream live audio

1. **Set API key**: Export `ASSEMBLYAI_API_KEY` environment variable.
2. **Choose model**: Default is `universal-3-6-pro` (recommended). Use `universal-3-5-pro` or `universal-streaming-multilingual` for cost savings.
3. **Create transcriber**: Instantiate `RealTimeTranscriber` (Python) or `streaming.transcriber()` (JavaScript).
4. **Connect**: Call `client.connect(RealTimeParameters(...))` to open WebSocket.
5. **Stream audio**: Send audio chunks via `client.stream(chunk)` as they arrive.
6. **Handle events**: Listen for `Turn` events (partial and final transcripts), `Begin` (session opened), `Termination` (session closed).
7. **Close session**: Call `client.disconnect(terminate=True)` when done; finalizes open turn and closes connection.
8. **Billing note**: Charged for entire session duration (time connection open), not audio sent. Always close sessions.

### Synthesize speech with TTS

1. **Set API key**: Export `ASSEMBLYAI_API_KEY` environment variable.
2. **Create session**: Call `client.tts.session(voiceId="aai-kyla", language="english")`.
3. **Connect**: Call `session.connect()` to open WebSocket and load voice.
4. **Send text**: Call `session.speak(text)` to synthesize; returns async iterator of audio chunks.
5. **Play audio**: Write chunks to file or stream to playback device.
6. **Keep alive**: Call `session.keepAlive()` during idle periods to prevent timeout.
7. **Close**: Call `session.close()` when conversation ends.

### Extract insights from transcript

1. **Transcribe first**: Get transcript ID from pre-recorded or streaming transcription.
2. **Choose features**: Enable on transcription request (`entity_detection=True`, `sentiment_analysis=True`, etc.) or call Speech Understanding API post-transcription.
3. **Access results**: Utterances include `sentiment`, `entities`, `speaker` (if diarization enabled), etc.
4. **Use LLM Gateway**: For complex analysis, send transcript to LLM Gateway with `POST /v1/chat/completions` for reasoning.

## Common gotchas

- **API key in client-side code**: Never ship API key to browser. Use temporary tokens (`POST /v1/streaming/token`) for client-side streaming.
- **File upload format**: Use `curl --data-binary @file` (raw bytes), not `-d` or JSON body, or transcoding fails with "File type application/json" error.
- **Polling rate limit**: 60 GET requests per minute per transcript. Use webhooks for high-volume jobs or jittered polling.
- **Streaming session billing**: Charged for entire session duration (time connection open), not audio sent. Idle time counts. Always send `Terminate` message and close connection.
- **Language detection misidentification**: Accented speech can be detected as wrong language (e.g., accented English detected as Finnish). Set `language_code` manually if auto-detection fails.
- **Speaker labels require diarization**: Enable `speaker_labels=True` to split utterances by speaker; without it, you get one continuous transcript.
- **Transcript ID required for support**: Always log `transcript.id` with timestamp and region; support needs it to look up requests.
- **Webhook timeout**: Endpoint must return 2xx within 10 seconds or AssemblyAI retries up to 10 times. 4xx stops retries.
- **Model fallback**: If language not supported by chosen model, system falls back to next in `speech_models` array. Check `speech_model_used` in response.
- **Temporary token expiration**: Temporary tokens expire; regenerate before session starts if reusing old token.

## Verification checklist

Before submitting work:

- [ ] API key set as environment variable (`ASSEMBLYAI_API_KEY`), not hardcoded
- [ ] Transcript ID logged for every request (required for debugging)
- [ ] Error handling checks `transcript.status == "error"` and includes error message
- [ ] Streaming sessions closed with `terminate=True` to finalize open turn
- [ ] Webhooks return 2xx status within 10 seconds (or polling used instead)
- [ ] File uploads use `--data-binary` (raw bytes), not JSON or file-path string
- [ ] Language code set manually if auto-detection unreliable (accented speech)
- [ ] Speaker labels enabled if per-speaker utterances needed
- [ ] Rate limits respected: 60 polling requests/min per transcript, 20k HTTP/5min total
- [ ] Model selection matches use case (U3.5 Pro for accuracy, U2 for 99 languages, streaming model for real-time)
- [ ] Temporary tokens regenerated before expiration if using client-side streaming

## Resources

- **Comprehensive page listing**: https://assemblyai.com/docs/llms.txt
- **Pre-recorded STT quickstart**: https://assemblyai.com/docs/pre-recorded-audio/getting-started/transcribe-an-audio-file
- **Streaming STT quickstart**: https://assemblyai.com/docs/streaming/getting-started/transcribe-streaming-audio
- **Models and pricing**: https://assemblyai.com/docs/getting-started/models

---

> For additional documentation and navigation, see: https://assemblyai.com/docs/llms.txt