Set up DeepInfra with an AI agent

name: deepinfra
description: Use when building AI applications with LLMs, embeddings, image
  generation, or speech models. Reach for this skill when integrating inference
  APIs, deploying private models, running batch jobs, or working with frameworks
  like LangChain, CrewAI, or Anthropic SDK.
metadata:
  mintlify-proj: deepinfra
  version: "1.0"

DeepInfra Skill

Product summary

DeepInfra is an AI inference cloud providing OpenAI-compatible APIs for 100+ open-source models (LLMs, embeddings, vision, image generation, speech, video). Agents use it to run inference at competitive per-token pricing, deploy private models on dedicated GPUs, or process large batches asynchronously at 20% discount.

Key endpoints:

• Chat completions: https://api.deepinfra.com/v1/openai/chat/completions
• Embeddings: https://api.deepinfra.com/v1/openai/embeddings
• Batch API: https://api.deepinfra.com/v1/openai/batches
• Image generation: https://api.deepinfra.com/v1/openai/images/generations

Authentication: Bearer token in Authorization header. Get API keys from dashboard at https://deepinfra.com/dash/api_keys. Keys support scopes: inference, models:read, models, batch, compute:sandbox, compute:agents, billing:read, etc.

Primary docs: https://docs.deepinfra.com

When to use

• Real-time inference: Call chat completions, embeddings, or image generation APIs for immediate responses
• Batch processing: Submit 50,000+ requests in a JSONL file for asynchronous processing at 20% discount (up to 24h completion)
• Private deployments: Deploy fine-tuned LLMs or LoRA adapters on dedicated A100/H100/H200/B200/B300 GPUs with autoscaling
• Multi-model fallback: Use the models parameter to specify fallback models when primary model is at capacity
• Framework integration: Use with LangChain, CrewAI, Anthropic SDK, LlamaIndex, or AI SDK via OpenAI-compatible interface
• Cost optimization: Use service_tier: "flex" for non-urgent work (20% discount, best-effort), or service_tier: "priority" for latency-critical requests (50% premium)
• Long-context applications: Leverage prompt caching with prompt_cache_key and prompt_cache_options to reduce token costs on repeated prefixes

Quick reference

Essential API parameters

Parameter	Use	Notes
model	Specify model	Format: namespace/model-name, MODEL:VERSION, or deploy_id:DEPLOY_ID
messages	Chat input	Array of {role, content} objects; roles: system, user, assistant
stream	Enable streaming	Set true for token-by-token SSE responses
temperature	Randomness	0–2; default 1.0; lower = deterministic, higher = creative
max_tokens	Output limit	Model-dependent; hard cap 16384 for most models
service_tier	Pricing tier	"priority" (50% premium, faster), "flex" (20% discount, best-effort)
fail_fast	Capacity handling	true = reject with 429 immediately; false = queue (default)
tools	Function calling	Array of tool definitions for agentic workflows
response_format	Output format	{"type": "json_object"} or {"type": "json_schema", "json_schema": {...}}
prompt_cache_key	Cache reuse	Tag request for KV cache reuse across sessions

Common model patterns

# Basic chat completion
client = OpenAI(
    api_key="sk-di-...",
    base_url="https://api.deepinfra.com/v1/openai"
)
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash-0731",
    messages=[{"role": "user", "content": "Hello"}]
)

# Streaming
for chunk in client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash-0731",
    messages=[...],
    stream=True
):
    print(chunk.choices[0].delta.content, end="")

# Embeddings
response = client.embeddings.create(
    model="Qwen/Qwen3-Embedding-8B",
    input="Text to embed"
)

# Batch (JSONL file)
batch_file = client.files.create(
    file=open("requests.jsonl", "rb"),
    purpose="batch"
)
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

Rate limits and quotas

• Concurrent requests: Per-model limit (default varies); request increase via API
• Tokens per minute (TPM): Per-model limit; check with GET /v1/me/rate_limit
• Batch limits: 50,000 requests/file, 200 MB file size, 100 concurrent batches
• Sandbox limits: 25 active sandboxes per account
• HTTP 429: Retry after brief delay; model may be busy even under limit

File and deployment operations

# List deployments
curl https://api.deepinfra.com/v1/deploy/list \
  -H "Authorization: Bearer $DEEPINFRA_API_KEY"

# Get deployment status
curl https://api.deepinfra.com/v1/deploy/{deploy_id} \
  -H "Authorization: Bearer $DEEPINFRA_API_KEY"

# Create deployment (via dashboard or API)
# POST /v1/deploy/create with model, GPU type, scaling config

Decision guidance

Scenario	Use	Why
Latency-sensitive, low volume	service_tier: "priority" or default	Pay 50% premium for guaranteed fast response
Non-urgent, high volume	service_tier: "flex" or Batch API	Save 20% on flex, 20% on batch; accept delays
50,000+ requests	Batch API	Asynchronous, 20% discount, no rate limit impact
Real-time inference	Chat/Embeddings API	Immediate response, per-token billing
Private data/custom model	Private deployment	Dedicated GPU, data isolation, autoscaling
Repeated prompts	Prompt caching with prompt_cache_key	Reduce input token cost by 90% on cache hits
Model at capacity	models: [fallback1, fallback2]	Server-side failover; first model to accept serves
Agentic workflows	Tool calling + streaming	Function definitions + token-by-token responses

Workflow

1. Set up authentication

• Create API key at https://deepinfra.com/dash/api_keys
• Choose scopes: inference (required for inference), models:read, batch, etc.
• Export as environment variable: export DEEPINFRA_API_KEY=sk-di-...

2. Choose inference method

• Real-time: Use chat/embeddings/image endpoints for immediate responses
• Batch: Prepare JSONL file, upload, create batch job, poll for results
• Streaming: Set stream: true for token-by-token SSE responses

3. Select model

• Browse catalog at https://deepinfra.com/models
• Use exact model ID (case-sensitive): namespace/model-name
• Check context window, pricing, and supported features (vision, tool calling, etc.)

4. Make request

• Set base_url to https://api.deepinfra.com/v1/openai
• Pass model name, messages, and parameters
• Handle streaming or poll for batch completion

5. Monitor and optimize

• Check usage with GET /v1/me/rate_limit
• Use prompt_cache_key for repeated prompts
• Set spending limits in dashboard to prevent runaway costs
• Use fail_fast: true for latency-sensitive applications

Common gotchas

• Model names are case-sensitive: deepseek-ai/DeepSeek-V4-Flash-0731 ≠ DeepSeek-ai/deepseek-v4-flash-0731
• Batch endpoint mismatch: All requests in a batch must use the same endpoint (/v1/chat/completions, /v1/completions, or /v1/embeddings); mixing causes validation error
• Private deployment costs: Billed per GPU-hour, not per token. Forgetting to stop a deployment can cost $256+ over a weekend
• Rate limit 429 errors: Retry with backoff; occasional 429s occur even under limit when model is busy (autoscaling kicks in)
• Scoped JWT expiration: Default 1 year; set expires_delta or expires_at to limit lifetime
• Prompt cache key reuse: Use session-scoped keys like userid-sessionid to maximize cache hits; different keys don't share cache
• Batch file size: Max 200 MB; 50,000 requests per file; larger jobs require multiple batches
• Tool calling accuracy: Not all models support tool calling equally; check model page for support
• Vision models: Require image_url in message content; some models support detail: "low" or "high" for token optimization
• Streaming + usage: Usage stats only in final chunk; don't rely on intermediate chunks for billing
• Flex tier rejection: May wait up to 10 minutes or return 429; use for retryable workloads only
• Unrestricted API keys: Avoid; prefer scoped keys with only needed permissions

Verification checklist

Before submitting work with DeepInfra:

• API key has correct scopes for the operation (e.g., inference for chat, batch for batch jobs)
• Model name is exact and case-sensitive
• Base URL is https://api.deepinfra.com/v1/openai (or appropriate endpoint)
• For batch: all requests use same endpoint, custom_id is unique per request, file is valid JSONL
• For streaming: handling finish_reason and usage in final chunk
• For private deployment: spending limit set in dashboard, deployment stopped after testing
• For tool calling: model supports tools (check catalog), tool definitions are valid JSON schema
• For vision: images passed as image_url with valid URL or base64 data URI
• For prompt caching: using consistent prompt_cache_key within session
• Error handling: retrying on 429, checking error.code for specific failures
• Rate limits: not exceeding per-model concurrent request or TPM limits

Resources

Comprehensive navigation: https://docs.deepinfra.com/llms.txt

Critical pages:

• Chat Completions API — OpenAI-compatible interface for LLMs
• Batch API — Asynchronous processing at 20% discount
• Authentication & Scopes — API key management and permissions

---

For additional documentation and navigation, see: https://docs.deepinfra.com/llms.txt