Set up Fireworks AI with an AI agent

name: public
description: Use when building LLM applications with serverless or on-demand
  inference, fine-tuning models, managing deployments, or automating ML
  workflows. Reach for this skill when working with open-source models,
  deploying custom models, running batch jobs, or optimizing model performance
  and costs.
metadata:
  mintlify-proj: public
  version: "1.0"

Fireworks AI Skill

Product summary

Fireworks AI is a platform for fast inference and training of open-source models. It offers two primary inference paths: Serverless (pay-per-token, shared infrastructure, instant access to 100+ models) and On-demand (dedicated GPUs, custom models, predictable performance). Use the REST API (OpenAI/Anthropic-compatible), Python SDK, or firectl CLI to query models, manage deployments, fine-tune models, and run batch jobs. Primary docs: https://docs.fireworks.ai

Key files and commands:

• API endpoint: https://api.fireworks.ai/inference/v1 (OpenAI-compatible)
• Authentication: FIREWORKS_API_KEY environment variable or --api-key flag
• CLI tool: firectl (sign in with firectl signin, manage deployments/models/training)
• Python SDK: from fireworks import Fireworks (drop-in replacement for OpenAI SDK)
• Model naming: accounts/fireworks/models/<MODEL_NAME> (serverless) or accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID> (on-demand)

When to use

Reach for this skill when:

• Prototyping quickly — Use Serverless with popular models (GLM, Llama, Deepseek, etc.) for immediate access without infrastructure setup
• Deploying to production — Create on-demand deployments on dedicated GPUs for predictable latency, throughput, and cost at scale
• Fine-tuning models — Train adapters (LoRA) or full models on your data using Managed Training or the Training API
• Running batch jobs — Process large datasets asynchronously with the Batch API at 50% of standard rates
• Managing infrastructure — Use firectl or the REST API to create, scale, and monitor deployments programmatically
• Optimizing costs — Choose between serverless (per-token), on-demand (per-GPU-hour), or batch (discounted) billing models
• Using advanced features — Leverage prompt caching, function calling, structured outputs, vision models, or reasoning

Quick reference

Authentication & setup

Task	Command
Create API key	Visit https://app.fireworks.ai/settings/users/api-keys
Export API key	export FIREWORKS_API_KEY="your_key_here"
Sign in with CLI	firectl signin
Check CLI auth	firectl whoami

Model naming conventions

Use case	Format	Example
Serverless model	accounts/fireworks/models/<MODEL_NAME>	accounts/fireworks/models/glm-5p3-flash
On-demand deployment	accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>	accounts/alice/deployments/12345678
LoRA on serverless	accounts/fireworks/models/<LORA_NAME>	Load via Specialized Model Inference API
LoRA on deployment	<MODEL_NAME>#<DEPLOYMENT_NAME>	accounts/fireworks/models/llama-v3p3-70b#my-lora

Essential CLI commands

Task	Command
List deployments	firectl deployment list
Create deployment	firectl deployment create accounts/fireworks/models/<MODEL> --deployment-shape default --wait
Get deployment status	firectl deployment get <DEPLOYMENT_ID>
Scale deployment	firectl deployment scale <DEPLOYMENT_ID> --min-replica-count 1 --max-replica-count 5
Delete deployment	firectl deployment delete <DEPLOYMENT_ID>
List models	firectl model list
Upload custom model	firectl model create <MODEL_ID> /path/to/model/
Create SFT job	firectl supervised-fine-tuning-job create --base-model <MODEL> --dataset <DATASET_ID>
List training jobs	firectl supervised-fine-tuning-job list

Python SDK essentials

from fireworks import Fireworks

client = Fireworks()

# Serverless inference
response = client.chat.completions.create(
    model="accounts/fireworks/models/glm-5p3-flash",
    messages=[{"role": "user", "content": "Hello"}]
)

# On-demand inference
response = client.chat.completions.create(
    model="accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>",
    messages=[{"role": "user", "content": "Hello"}]
)

# Streaming
stream = client.chat.completions.create(
    model="accounts/fireworks/models/glm-5p3-flash",
    messages=[{"role": "user", "content": "Tell a story"}],
    stream=True
)
for chunk in stream:
    print(chunk.choices[0].delta.content, end="")

# Structured output (JSON)
response = client.chat.completions.create(
    model="accounts/fireworks/models/glm-5p3-flash",
    messages=[{"role": "user", "content": "Extract name and age from: John is 30"}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "person",
            "schema": {
                "type": "object",
                "properties": {"name": {"type": "string"}, "age": {"type": "number"}},
                "required": ["name", "age"]
            }
        }
    }
)

Decision guidance

Serverless vs. On-demand

Factor	Serverless	On-demand
Billing	Per token (input, cached, output)	Per GPU-hour
Setup time	Instant (no deployment)	5-10 minutes
Model selection	100+ popular models only	Any model (including custom)
Latency	Shared infrastructure, variable	Dedicated GPUs, predictable
Throughput	Adaptive rate limits (TPM)	Limited by replica count
Cost at scale	Higher per token	Lower per token at high volume
Use case	Prototyping, low-volume, variable load	Production, high-volume, consistent load

Decision: Start with Serverless for prototyping. Move to On-demand when you need custom models, predictable latency, or high sustained throughput.

Training methods

Method	Data required	Runs on	Cost model	Best for
Managed SFT	100+ input/output pairs	Managed	Per token	Quick fine-tuning, no code
Managed DPO	100+ chosen/rejected pairs	Managed	Per token	Preference optimization
Training API (SFT)	100+ input/output pairs	Serverless or Dedicated	Per token or per GPU-hour	Custom training loops
Training API (RL)	100+ prompts + scoring function	Dedicated	Per GPU-hour	Reasoning, agent tasks

Decision: Use Managed Training for simplicity. Use Training API when you need custom loss functions, reinforcement learning, or inference in the loop.

Batch vs. real-time inference

Aspect	Real-time (Chat API)	Batch (Batch API)
Latency	Immediate	Hours to days
Throughput	Per-request	Bulk processing
Billing	Standard rates	50% discount on input/output
Use case	Interactive apps, APIs	Evaluation, data processing, bulk labeling

Decision: Use Chat API for interactive workloads. Use Batch API for non-time-sensitive bulk processing to save 50% on costs.

Workflow

1. Prototype with Serverless

1. Get an API key — Visit https://app.fireworks.ai/settings/users/api-keys and create one
2. Export the key — export FIREWORKS_API_KEY="your_key"
3. Choose a model — Browse https://fireworks.ai/models or check recommended models
4. Make your first call — Use Python SDK, OpenAI SDK, or curl with accounts/fireworks/models/<MODEL_NAME>
5. Test features — Try streaming, function calling, structured outputs, or vision models
6. Monitor usage — Check response headers for token counts and rate limits

2. Deploy to production (on-demand)

1. Check deployment shapes — firectl deployment-shape-version match --model <MODEL_NAME> to see available hardware
2. Create deployment — firectl deployment create accounts/fireworks/models/<MODEL> --deployment-shape default --wait
3. Save deployment ID — Extract from output (format: accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>)
4. Query the deployment — Use the deployment ID in your model parameter
5. Configure autoscaling — firectl deployment update <DEPLOYMENT_ID> --min-replica-count 1 --max-replica-count 10
6. Monitor performance — Use firectl deployment get <DEPLOYMENT_ID> to check replica count and status

3. Fine-tune a model

1. Prepare data — Format as JSONL with messages (chat) or prompt/completion (completion) fields
2. Create dataset — firectl dataset create <DATASET_ID> /path/to/data.jsonl
3. Start training — firectl supervised-fine-tuning-job create --base-model <MODEL> --dataset <DATASET_ID> --output-model <OUTPUT_NAME>
4. Monitor job — firectl supervised-fine-tuning-job get <JOB_ID> (check status, loss curves)
5. Deploy trained model — Create an on-demand deployment with the trained model, or load on serverless if base model is supported
6. Query trained model — Use accounts/<ACCOUNT_ID>/models/<TRAINED_MODEL_ID> or the deployment ID

4. Run batch inference

1. Prepare input dataset — JSONL with messages or prompt fields
2. Create input dataset — firectl dataset create <INPUT_DATASET_ID> /path/to/input.jsonl
3. Create output dataset — firectl dataset create <OUTPUT_DATASET_ID> (empty, for results)
4. Submit batch job — firectl batch-inference-job create --model <MODEL> --input-dataset <INPUT_DATASET_ID> --output-dataset <OUTPUT_DATASET_ID>
5. Monitor progress — firectl batch-inference-job get <JOB_ID> (check percent complete, token counts)
6. Download results — firectl dataset download <OUTPUT_DATASET_ID> /local/path/

Common gotchas

• Missing --deployment-shape — Always pass --deployment-shape default or a specific shape when creating deployments. Shapeless deployments skip validation and are the most common cause of creation failures.
• Region placement is immutable — Set --region at deployment creation time. To change it, delete and recreate the deployment.
• Serverless model lifecycle — Models may be updated or deprecated with 2+ weeks notice. For long-term stability, use on-demand deployments.
• Rate limits on serverless — Limits are adaptive and based on spend tier. Check response headers (X-Ratelimit-Limit-Tokens-*) to see your current TPM ceiling.
• Prompt caching requires session affinity — Pass x-session-affinity header (or OpenAI user field) with a stable identifier to maximize cache hits on repeated prompts.
• Batch API billing — Batch jobs are billed at 50% of standard rates, but they run asynchronously (12–72 hour window). Don't use for latency-sensitive workloads.
• LoRA deployment format — On-demand deployments use <MODEL>#<DEPLOYMENT_NAME> format. Serverless uses the Specialized Model Inference API (load/unload endpoints).
• API key expiration — API keys expire after 1 year by default. Regenerate before expiration to avoid service interruption.
• Deployment scaling to zero — Deployments scale to zero after 1 hour of inactivity. Requests return 503 while scaling up; implement retry logic.
• Training data format — Ensure JSONL is valid (one JSON object per line). Malformed data causes validation failures before training starts.

Verification checklist

Before submitting work with Fireworks:

• API key is set — echo $FIREWORKS_API_KEY returns a non-empty value
• Model name is correct — Verify format: accounts/fireworks/models/<NAME> (serverless) or accounts/<ACCOUNT_ID>/deployments/<ID> (on-demand)
• Deployment is ready — firectl deployment get <ID> shows state: READY and ready_replica_count > 0
• Rate limits are understood — Check response headers or account quotas for TPM/RPM limits
• Billing model is chosen — Confirm serverless (per-token), on-demand (per-GPU-hour), or batch (50% discount)
• Prompt caching is configured (if using repeated prompts) — Pass x-session-affinity header for cache hits
• Error handling is in place — Implement retries for 429 (rate limit), 503 (scaling), and transient errors
• Costs are estimated — Use the training cost estimator for fine-tuning jobs; calculate token costs for inference
• Data is validated — For training, ensure JSONL format and required fields (messages or prompt/completion)
• Monitoring is set up — Log token usage, latency, and error rates for production workloads

Resources

Comprehensive navigation: https://docs.fireworks.ai/llms.txt

Critical documentation:

• Serverless Quickstart — Make your first API call in minutes
• On-demand Deployments — Create and manage dedicated GPU deployments
• Fine-tuning Overview — Train models with Managed Training or the Training API
• API Reference — Full REST API and SDK documentation
• Batch Inference — Run async jobs at 50% discount
• Prompt Caching — Optimize repeated prompts with caching
• Structured Outputs — Get reliable JSON responses
• Function Calling — Connect models to tools and APIs

---

For additional documentation and navigation, see: https://docs.fireworks.ai/llms.txt