Set up Portkey with an AI agent

name: portkey
description: Use when building AI applications that need unified access to 250+
  LLM providers, implementing reliability patterns (fallbacks, retries, load
  balancing), observability (logs, analytics, tracing), guardrails (PII
  detection, content filtering), prompt management, or MCP server integration.
  Essential for production AI systems requiring cost control, latency
  optimization, and security.
metadata:
  mintlify-proj: portkey
  version: "1.0"

Portkey AI Gateway Skill

Product Summary

Portkey is a unified AI Gateway that routes requests to 250+ LLM providers (OpenAI, Anthropic, Google, Azure, Bedrock, etc.) through a single API. It provides observability (logs, traces, analytics), reliability patterns (fallbacks, retries, load balancing, circuit breakers), guardrails (PII redaction, content filtering), prompt management, and MCP server integration. Agents use Portkey to build resilient, observable, and secure AI applications without vendor lock-in.

Key files and concepts:

• Configs (product/ai-gateway/configs): JSON objects defining routing, fallbacks, caching, retries
• Integrations (product/model-catalog): Stored provider credentials (OpenAI, Anthropic, etc.)
• Guardrails (product/guardrails): Input/output validation and security checks
• Observability (product/observability): Logs, traces, analytics, cost tracking
• SDKs: Python (portkey_ai), Node.js (portkey-ai), or OpenAI SDK with Portkey headers
• REST API: https://api.portkey.ai/v1/ (OpenAI-compatible)
• Admin API: https://api.portkey.ai/admin/ for managing configs, guardrails, integrations

Primary docs: https://docs.portkey.ai/docs

---

When to Use

Reach for this skill when:

• Routing LLM requests: Agent needs to send requests to multiple providers or models with fallback logic
• Building resilient AI apps: Implementing automatic retries, fallbacks, load balancing, or circuit breakers
• Cost and latency optimization: Caching responses, tracking costs per request, or A/B testing models
• Security and compliance: Detecting/redacting PII, filtering harmful content, enforcing policies
• Observability: Tracking request logs, latency, token usage, errors, or building analytics dashboards
• Prompt management: Versioning, testing, and deploying prompts across teams
• Multi-tenant AI: Isolating budgets, rate limits, and guardrails per workspace or user
• MCP integration: Connecting AI models to external tools via Model Context Protocol servers
• Provider migration: Switching between providers without code changes (via configs)

---

Quick Reference

Authentication & Setup

Method	Use Case
Portkey SDK	Recommended for Python/Node.js; handles auth automatically
OpenAI SDK + Portkey headers	Use existing OpenAI code; add x-portkey-api-key header
REST API + headers	Direct HTTP calls; include x-portkey-api-key, x-portkey-provider
Config on API key	Attach default config to API key; applies to all requests with that key

Essential Headers

x-portkey-api-key: YOUR_PORTKEY_API_KEY
x-portkey-provider: @provider-slug  (e.g., @openai-prod)
x-portkey-config: pc-xxxxx  (config ID or JSON object)
x-portkey-metadata: {"user_id": "123"}  (custom tracking)

Config Structure (JSON)

{
  "strategy": { "mode": "fallback|loadbalance|conditional" },
  "targets": [
    {
      "provider": "@openai-prod",
      "override_params": { "model": "gpt-4o" },
      "default_params": { "temperature": 0.7 },
      "drop_params": ["logprobs"]
    }
  ],
  "retry": { "attempts": 3, "on_status_codes": [429, 500, 502, 503] },
  "cache": { "mode": "simple|semantic", "ttl": 3600 },
  "input_guardrails": ["guardrail-id-xxx"],
  "output_guardrails": ["guardrail-id-yyy"]
}

Common CLI/SDK Patterns

Python:

from portkey_ai import Portkey

portkey = Portkey(
    api_key="YOUR_PORTKEY_API_KEY",
    provider="@openai-prod",
    config="pc-xxxxx"  # Optional: config ID
)

response = portkey.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}]
)

Node.js:

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: "YOUR_PORTKEY_API_KEY",
    provider: "@openai-prod",
    config: "pc-xxxxx"
});

const response = await portkey.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: "Hello" }]
});

REST API:

curl https://api.portkey.ai/v1/chat/completions \
  -H "x-portkey-api-key: YOUR_PORTKEY_API_KEY" \
  -H "x-portkey-provider: @openai-prod" \
  -H "x-portkey-config: pc-xxxxx" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'

---

Decision Guidance

When to Use X vs Y

Decision	Use X When	Use Y When
Fallback vs Load Balance	One provider fails → try another	Distribute load across healthy providers
Simple vs Semantic Cache	Exact prompt match	Similar prompts (e.g., "What is AI?" vs "Define AI?")
Async vs Sync Guardrails	Logging only; no latency impact	Blocking requests on guardrail failure
Config on key vs per-request	All requests use same routing	Different requests need different strategies
Portkey SDK vs OpenAI SDK	Full feature access (MCP, advanced routing)	Minimal code changes; OpenAI-compatible
Input vs Output Guardrails	Validate user prompts before LLM	Validate LLM responses before user
Conditional Routing	Route based on model, user, metadata	Fixed provider/model routing

---

Workflow

1. Set Up Provider Credentials (One-Time)

1. Go to Model Catalog → Integrations
2. Create integration: select provider (OpenAI, Anthropic, etc.), paste API key
3. Create provider: name it (e.g., @openai-prod), select integration, set budget/rate limits
4. Note the provider slug for use in configs

2. Create a Config (Routing Logic)

1. Go to Configs → Create
2. Define strategy: fallback, load balance, or conditional routing
3. Add targets: specify providers/models, override params, add guardrails
4. Save and copy config ID (e.g., pc-xxxxx)

3. Create Guardrails (Optional Security)

1. Go to Guardrails → Create
2. Add checks: PII detection, regex match, JSON schema, etc.
3. Set actions: async/sync, deny on fail, send feedback
4. Save and copy guardrail ID
5. Add guardrail ID to config's input_guardrails or output_guardrails

4. Make Requests

Option A: Attach config to API key (default for all requests)

• Go to API Keys, select key, set Default Config
• All requests with that key use the config automatically

Option B: Pass config per-request

• Python: portkey.with_options(config="pc-xxxxx").chat.completions.create(...)
• Node.js: portkey.chat.completions.create(..., {config: "pc-xxxxx"})
• REST: Add x-portkey-config: pc-xxxxx header

5. Monitor & Debug

1. Go to Logs → filter by config ID, trace ID, or status
2. Click log entry → view full request/response, guardrail results, latency
3. Go to Analytics → view cost, latency, error rate, feedback trends
4. Use Traces to follow multi-step requests (e.g., fallback chain)

---

Common Gotchas

• Config not applying: Verify config ID is correct and API key has permission to use it. Request-level config overrides key-level default.
• Fallback not triggering: Check on_status_codes — fallback only triggers on listed codes (default: all non-2xx). A 400 won't trigger if only [429, 500] are listed.
• Guardrails blocking all requests: Ensure guardrail is set to async: true or deny: false during testing. deny: true + async: false blocks on any failure.
• PII redaction not working: Redaction is irreversible; verify PII categories are enabled in guardrail. Original data is not stored by Portkey.
• High latency: Portkey adds 20-40ms. If latency is higher, check: (1) provider response time, (2) guardrail execution time, (3) network location.
• Costs not tracking: Ensure metadata is sent with requests (x-portkey-metadata header). Costs are calculated from token counts in provider responses.
• Model not found: Use format @provider-slug/model-name (e.g., @openai-prod/gpt-4o). Verify provider slug exists in Model Catalog.
• Streaming with guardrails: Output guardrails on streaming responses are informational only; fallback/retry don't trigger. Set x-portkey-strict-open-ai-compliance: false to see guardrail results in stream.
• Rate limit errors: Use load balancing across multiple API keys or set on_status_codes: [429] in fallback to switch providers on rate limit.
• Prompt versioning confusion: "Save" creates a new version; "Publish" marks it as production. Always publish before using in production.

---

Verification Checklist

Before submitting work:

• Config created and saved: Verify config ID in UI or via Admin API
• Provider credentials stored: Integration exists in Model Catalog with valid API key
• Guardrails tested: If using guardrails, test with async: true first, then async: false
• Request headers correct: x-portkey-api-key, x-portkey-provider, x-portkey-config present
• Fallback chain tested: Simulate primary failure; verify fallback triggers on expected status codes
• Logs visible: Make test request, check Logs page for entry with correct config ID
• Cost tracking enabled: Metadata sent with requests; Analytics shows usage
• Latency acceptable: Check log latency; if >100ms, investigate provider or guardrail performance
• Guardrail results visible: If using guardrails, check log entry for hook_results object
• Production config locked: Use enforce-saved-only-config to prevent ad-hoc config changes in production

---

Resources

Comprehensive navigation: https://docs.portkey.ai/docs/llms.txt

Critical pages:

1. AI Gateway Overview — Features, routing strategies
2. Configs Documentation — Config schema, examples, passthrough targets
3. Observability & Logs — Viewing requests, debugging, cost tracking
4. Guardrails — Creating checks, actions, integration with configs
5. Model Catalog — Managing integrations, providers, budgets, rate limits
6. Admin API Reference — Programmatic config, guardrail, integration management
7. SDK Reference — Python, Node.js, C# SDK usage

---

For additional documentation and navigation, see: https://docs.portkey.ai/docs/llms.txt