---
name: deepinfra
description: Use when building AI applications with LLMs, embeddings, image
  generation, or speech models. Reach for this skill when integrating inference
  APIs, deploying private models, running batch jobs, or working with frameworks
  like LangChain, CrewAI, or Anthropic SDK.
metadata:
  mintlify-proj: deepinfra
  version: "1.0"
---

# DeepInfra Skill

## Product summary

DeepInfra is an AI inference cloud providing OpenAI-compatible APIs for 100+ open-source models (LLMs, embeddings, vision, image generation, speech, video). Agents use it to run inference at competitive per-token pricing, deploy private models on dedicated GPUs, or process large batches asynchronously at 20% discount.

**Key endpoints:**
- Chat completions: `https://api.deepinfra.com/v1/openai/chat/completions`
- Embeddings: `https://api.deepinfra.com/v1/openai/embeddings`
- Batch API: `https://api.deepinfra.com/v1/openai/batches`
- Image generation: `https://api.deepinfra.com/v1/openai/images/generations`

**Authentication:** Bearer token in `Authorization` header. Get API keys from dashboard at `https://deepinfra.com/dash/api_keys`. Keys support scopes: `inference`, `models:read`, `models`, `batch`, `compute:sandbox`, `compute:agents`, `billing:read`, etc.

**Primary docs:** https://docs.deepinfra.com

## When to use

- **Real-time inference:** Call chat completions, embeddings, or image generation APIs for immediate responses
- **Batch processing:** Submit 50,000+ requests in a JSONL file for asynchronous processing at 20% discount (up to 24h completion)
- **Private deployments:** Deploy fine-tuned LLMs or LoRA adapters on dedicated A100/H100/H200/B200/B300 GPUs with autoscaling
- **Multi-model fallback:** Use the `models` parameter to specify fallback models when primary model is at capacity
- **Framework integration:** Use with LangChain, CrewAI, Anthropic SDK, LlamaIndex, or AI SDK via OpenAI-compatible interface
- **Cost optimization:** Use `service_tier: "flex"` for non-urgent work (20% discount, best-effort), or `service_tier: "priority"` for latency-critical requests (50% premium)
- **Long-context applications:** Leverage prompt caching with `prompt_cache_key` and `prompt_cache_options` to reduce token costs on repeated prefixes

## Quick reference

### Essential API parameters

| Parameter | Use | Notes |
|-----------|-----|-------|
| `model` | Specify model | Format: `namespace/model-name`, `MODEL:VERSION`, or `deploy_id:DEPLOY_ID` |
| `messages` | Chat input | Array of `{role, content}` objects; roles: `system`, `user`, `assistant` |
| `stream` | Enable streaming | Set `true` for token-by-token SSE responses |
| `temperature` | Randomness | 0–2; default 1.0; lower = deterministic, higher = creative |
| `max_tokens` | Output limit | Model-dependent; hard cap 16384 for most models |
| `service_tier` | Pricing tier | `"priority"` (50% premium, faster), `"flex"` (20% discount, best-effort) |
| `fail_fast` | Capacity handling | `true` = reject with 429 immediately; `false` = queue (default) |
| `tools` | Function calling | Array of tool definitions for agentic workflows |
| `response_format` | Output format | `{"type": "json_object"}` or `{"type": "json_schema", "json_schema": {...}}` |
| `prompt_cache_key` | Cache reuse | Tag request for KV cache reuse across sessions |

### Common model patterns

```python
# Basic chat completion
client = OpenAI(
    api_key="sk-di-...",
    base_url="https://api.deepinfra.com/v1/openai"
)
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash-0731",
    messages=[{"role": "user", "content": "Hello"}]
)

# Streaming
for chunk in client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash-0731",
    messages=[...],
    stream=True
):
    print(chunk.choices[0].delta.content, end="")

# Embeddings
response = client.embeddings.create(
    model="Qwen/Qwen3-Embedding-8B",
    input="Text to embed"
)

# Batch (JSONL file)
batch_file = client.files.create(
    file=open("requests.jsonl", "rb"),
    purpose="batch"
)
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h"
)
```

### Rate limits and quotas

- **Concurrent requests:** Per-model limit (default varies); request increase via API
- **Tokens per minute (TPM):** Per-model limit; check with `GET /v1/me/rate_limit`
- **Batch limits:** 50,000 requests/file, 200 MB file size, 100 concurrent batches
- **Sandbox limits:** 25 active sandboxes per account
- **HTTP 429:** Retry after brief delay; model may be busy even under limit

### File and deployment operations

```bash
# List deployments
curl https://api.deepinfra.com/v1/deploy/list \
  -H "Authorization: Bearer $DEEPINFRA_API_KEY"

# Get deployment status
curl https://api.deepinfra.com/v1/deploy/{deploy_id} \
  -H "Authorization: Bearer $DEEPINFRA_API_KEY"

# Create deployment (via dashboard or API)
# POST /v1/deploy/create with model, GPU type, scaling config
```

## Decision guidance

| Scenario | Use | Why |
|----------|-----|-----|
| **Latency-sensitive, low volume** | `service_tier: "priority"` or default | Pay 50% premium for guaranteed fast response |
| **Non-urgent, high volume** | `service_tier: "flex"` or Batch API | Save 20% on flex, 20% on batch; accept delays |
| **50,000+ requests** | Batch API | Asynchronous, 20% discount, no rate limit impact |
| **Real-time inference** | Chat/Embeddings API | Immediate response, per-token billing |
| **Private data/custom model** | Private deployment | Dedicated GPU, data isolation, autoscaling |
| **Repeated prompts** | Prompt caching with `prompt_cache_key` | Reduce input token cost by 90% on cache hits |
| **Model at capacity** | `models: [fallback1, fallback2]` | Server-side failover; first model to accept serves |
| **Agentic workflows** | Tool calling + streaming | Function definitions + token-by-token responses |

## Workflow

### 1. Set up authentication
- Create API key at `https://deepinfra.com/dash/api_keys`
- Choose scopes: `inference` (required for inference), `models:read`, `batch`, etc.
- Export as environment variable: `export DEEPINFRA_API_KEY=sk-di-...`

### 2. Choose inference method
- **Real-time:** Use chat/embeddings/image endpoints for immediate responses
- **Batch:** Prepare JSONL file, upload, create batch job, poll for results
- **Streaming:** Set `stream: true` for token-by-token SSE responses

### 3. Select model
- Browse catalog at `https://deepinfra.com/models`
- Use exact model ID (case-sensitive): `namespace/model-name`
- Check context window, pricing, and supported features (vision, tool calling, etc.)

### 4. Make request
- Set `base_url` to `https://api.deepinfra.com/v1/openai`
- Pass model name, messages, and parameters
- Handle streaming or poll for batch completion

### 5. Monitor and optimize
- Check usage with `GET /v1/me/rate_limit`
- Use `prompt_cache_key` for repeated prompts
- Set spending limits in dashboard to prevent runaway costs
- Use `fail_fast: true` for latency-sensitive applications

## Common gotchas

- **Model names are case-sensitive:** `deepseek-ai/DeepSeek-V4-Flash-0731` ≠ `DeepSeek-ai/deepseek-v4-flash-0731`
- **Batch endpoint mismatch:** All requests in a batch must use the same endpoint (`/v1/chat/completions`, `/v1/completions`, or `/v1/embeddings`); mixing causes validation error
- **Private deployment costs:** Billed per GPU-hour, not per token. Forgetting to stop a deployment can cost $256+ over a weekend
- **Rate limit 429 errors:** Retry with backoff; occasional 429s occur even under limit when model is busy (autoscaling kicks in)
- **Scoped JWT expiration:** Default 1 year; set `expires_delta` or `expires_at` to limit lifetime
- **Prompt cache key reuse:** Use session-scoped keys like `userid-sessionid` to maximize cache hits; different keys don't share cache
- **Batch file size:** Max 200 MB; 50,000 requests per file; larger jobs require multiple batches
- **Tool calling accuracy:** Not all models support tool calling equally; check model page for support
- **Vision models:** Require `image_url` in message content; some models support `detail: "low"` or `"high"` for token optimization
- **Streaming + usage:** Usage stats only in final chunk; don't rely on intermediate chunks for billing
- **Flex tier rejection:** May wait up to 10 minutes or return 429; use for retryable workloads only
- **Unrestricted API keys:** Avoid; prefer scoped keys with only needed permissions

## Verification checklist

Before submitting work with DeepInfra:

- [ ] API key has correct scopes for the operation (e.g., `inference` for chat, `batch` for batch jobs)
- [ ] Model name is exact and case-sensitive
- [ ] Base URL is `https://api.deepinfra.com/v1/openai` (or appropriate endpoint)
- [ ] For batch: all requests use same endpoint, custom_id is unique per request, file is valid JSONL
- [ ] For streaming: handling `finish_reason` and usage in final chunk
- [ ] For private deployment: spending limit set in dashboard, deployment stopped after testing
- [ ] For tool calling: model supports tools (check catalog), tool definitions are valid JSON schema
- [ ] For vision: images passed as `image_url` with valid URL or base64 data URI
- [ ] For prompt caching: using consistent `prompt_cache_key` within session
- [ ] Error handling: retrying on 429, checking `error.code` for specific failures
- [ ] Rate limits: not exceeding per-model concurrent request or TPM limits

## Resources

**Comprehensive navigation:** https://docs.deepinfra.com/llms.txt

**Critical pages:**
- [Chat Completions API](https://docs.deepinfra.com/chat/overview) — OpenAI-compatible interface for LLMs
- [Batch API](https://docs.deepinfra.com/batch/introduction) — Asynchronous processing at 20% discount
- [Authentication & Scopes](https://docs.deepinfra.com/account/authentication) — API key management and permissions

---

> For additional documentation and navigation, see: https://docs.deepinfra.com/llms.txt