---
name: public
description: Use when building LLM applications with serverless or on-demand
  inference, fine-tuning models, managing deployments, or automating ML
  workflows. Reach for this skill when working with open-source models,
  deploying custom models, running batch jobs, or optimizing model performance
  and costs.
metadata:
  mintlify-proj: public
  version: "1.0"
---

# Fireworks AI Skill

## Product summary

Fireworks AI is a platform for fast inference and training of open-source models. It offers two primary inference paths: **Serverless** (pay-per-token, shared infrastructure, instant access to 100+ models) and **On-demand** (dedicated GPUs, custom models, predictable performance). Use the REST API (OpenAI/Anthropic-compatible), Python SDK, or `firectl` CLI to query models, manage deployments, fine-tune models, and run batch jobs. Primary docs: https://docs.fireworks.ai

**Key files and commands:**
- API endpoint: `https://api.fireworks.ai/inference/v1` (OpenAI-compatible)
- Authentication: `FIREWORKS_API_KEY` environment variable or `--api-key` flag
- CLI tool: `firectl` (sign in with `firectl signin`, manage deployments/models/training)
- Python SDK: `from fireworks import Fireworks` (drop-in replacement for OpenAI SDK)
- Model naming: `accounts/fireworks/models/<MODEL_NAME>` (serverless) or `accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>` (on-demand)

## When to use

Reach for this skill when:
- **Prototyping quickly** — Use Serverless with popular models (GLM, Llama, Deepseek, etc.) for immediate access without infrastructure setup
- **Deploying to production** — Create on-demand deployments on dedicated GPUs for predictable latency, throughput, and cost at scale
- **Fine-tuning models** — Train adapters (LoRA) or full models on your data using Managed Training or the Training API
- **Running batch jobs** — Process large datasets asynchronously with the Batch API at 50% of standard rates
- **Managing infrastructure** — Use `firectl` or the REST API to create, scale, and monitor deployments programmatically
- **Optimizing costs** — Choose between serverless (per-token), on-demand (per-GPU-hour), or batch (discounted) billing models
- **Using advanced features** — Leverage prompt caching, function calling, structured outputs, vision models, or reasoning

## Quick reference

### Authentication & setup

| Task | Command |
|------|---------|
| Create API key | Visit https://app.fireworks.ai/settings/users/api-keys |
| Export API key | `export FIREWORKS_API_KEY="your_key_here"` |
| Sign in with CLI | `firectl signin` |
| Check CLI auth | `firectl whoami` |

### Model naming conventions

| Use case | Format | Example |
|----------|--------|---------|
| Serverless model | `accounts/fireworks/models/<MODEL_NAME>` | `accounts/fireworks/models/glm-5p3-flash` |
| On-demand deployment | `accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>` | `accounts/alice/deployments/12345678` |
| LoRA on serverless | `accounts/fireworks/models/<LORA_NAME>` | Load via Specialized Model Inference API |
| LoRA on deployment | `<MODEL_NAME>#<DEPLOYMENT_NAME>` | `accounts/fireworks/models/llama-v3p3-70b#my-lora` |

### Essential CLI commands

| Task | Command |
|------|---------|
| List deployments | `firectl deployment list` |
| Create deployment | `firectl deployment create accounts/fireworks/models/<MODEL> --deployment-shape default --wait` |
| Get deployment status | `firectl deployment get <DEPLOYMENT_ID>` |
| Scale deployment | `firectl deployment scale <DEPLOYMENT_ID> --min-replica-count 1 --max-replica-count 5` |
| Delete deployment | `firectl deployment delete <DEPLOYMENT_ID>` |
| List models | `firectl model list` |
| Upload custom model | `firectl model create <MODEL_ID> /path/to/model/` |
| Create SFT job | `firectl supervised-fine-tuning-job create --base-model <MODEL> --dataset <DATASET_ID>` |
| List training jobs | `firectl supervised-fine-tuning-job list` |

### Python SDK essentials

```python
from fireworks import Fireworks

client = Fireworks()

# Serverless inference
response = client.chat.completions.create(
    model="accounts/fireworks/models/glm-5p3-flash",
    messages=[{"role": "user", "content": "Hello"}]
)

# On-demand inference
response = client.chat.completions.create(
    model="accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>",
    messages=[{"role": "user", "content": "Hello"}]
)

# Streaming
stream = client.chat.completions.create(
    model="accounts/fireworks/models/glm-5p3-flash",
    messages=[{"role": "user", "content": "Tell a story"}],
    stream=True
)
for chunk in stream:
    print(chunk.choices[0].delta.content, end="")

# Structured output (JSON)
response = client.chat.completions.create(
    model="accounts/fireworks/models/glm-5p3-flash",
    messages=[{"role": "user", "content": "Extract name and age from: John is 30"}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "person",
            "schema": {
                "type": "object",
                "properties": {"name": {"type": "string"}, "age": {"type": "number"}},
                "required": ["name", "age"]
            }
        }
    }
)
```

## Decision guidance

### Serverless vs. On-demand

| Factor | Serverless | On-demand |
|--------|-----------|-----------|
| **Billing** | Per token (input, cached, output) | Per GPU-hour |
| **Setup time** | Instant (no deployment) | 5-10 minutes |
| **Model selection** | 100+ popular models only | Any model (including custom) |
| **Latency** | Shared infrastructure, variable | Dedicated GPUs, predictable |
| **Throughput** | Adaptive rate limits (TPM) | Limited by replica count |
| **Cost at scale** | Higher per token | Lower per token at high volume |
| **Use case** | Prototyping, low-volume, variable load | Production, high-volume, consistent load |

**Decision:** Start with Serverless for prototyping. Move to On-demand when you need custom models, predictable latency, or high sustained throughput.

### Training methods

| Method | Data required | Runs on | Cost model | Best for |
|--------|---------------|---------|-----------|----------|
| **Managed SFT** | 100+ input/output pairs | Managed | Per token | Quick fine-tuning, no code |
| **Managed DPO** | 100+ chosen/rejected pairs | Managed | Per token | Preference optimization |
| **Training API (SFT)** | 100+ input/output pairs | Serverless or Dedicated | Per token or per GPU-hour | Custom training loops |
| **Training API (RL)** | 100+ prompts + scoring function | Dedicated | Per GPU-hour | Reasoning, agent tasks |

**Decision:** Use Managed Training for simplicity. Use Training API when you need custom loss functions, reinforcement learning, or inference in the loop.

### Batch vs. real-time inference

| Aspect | Real-time (Chat API) | Batch (Batch API) |
|--------|---------------------|-------------------|
| **Latency** | Immediate | Hours to days |
| **Throughput** | Per-request | Bulk processing |
| **Billing** | Standard rates | 50% discount on input/output |
| **Use case** | Interactive apps, APIs | Evaluation, data processing, bulk labeling |

**Decision:** Use Chat API for interactive workloads. Use Batch API for non-time-sensitive bulk processing to save 50% on costs.

## Workflow

### 1. Prototype with Serverless

1. **Get an API key** — Visit https://app.fireworks.ai/settings/users/api-keys and create one
2. **Export the key** — `export FIREWORKS_API_KEY="your_key"`
3. **Choose a model** — Browse https://fireworks.ai/models or check [recommended models](/guides/recommended-models)
4. **Make your first call** — Use Python SDK, OpenAI SDK, or curl with `accounts/fireworks/models/<MODEL_NAME>`
5. **Test features** — Try streaming, function calling, structured outputs, or vision models
6. **Monitor usage** — Check response headers for token counts and rate limits

### 2. Deploy to production (on-demand)

1. **Check deployment shapes** — `firectl deployment-shape-version match --model <MODEL_NAME>` to see available hardware
2. **Create deployment** — `firectl deployment create accounts/fireworks/models/<MODEL> --deployment-shape default --wait`
3. **Save deployment ID** — Extract from output (format: `accounts/<ACCOUNT_ID>/deployments/<DEPLOYMENT_ID>`)
4. **Query the deployment** — Use the deployment ID in your model parameter
5. **Configure autoscaling** — `firectl deployment update <DEPLOYMENT_ID> --min-replica-count 1 --max-replica-count 10`
6. **Monitor performance** — Use `firectl deployment get <DEPLOYMENT_ID>` to check replica count and status

### 3. Fine-tune a model

1. **Prepare data** — Format as JSONL with `messages` (chat) or `prompt`/`completion` (completion) fields
2. **Create dataset** — `firectl dataset create <DATASET_ID> /path/to/data.jsonl`
3. **Start training** — `firectl supervised-fine-tuning-job create --base-model <MODEL> --dataset <DATASET_ID> --output-model <OUTPUT_NAME>`
4. **Monitor job** — `firectl supervised-fine-tuning-job get <JOB_ID>` (check status, loss curves)
5. **Deploy trained model** — Create an on-demand deployment with the trained model, or load on serverless if base model is supported
6. **Query trained model** — Use `accounts/<ACCOUNT_ID>/models/<TRAINED_MODEL_ID>` or the deployment ID

### 4. Run batch inference

1. **Prepare input dataset** — JSONL with `messages` or `prompt` fields
2. **Create input dataset** — `firectl dataset create <INPUT_DATASET_ID> /path/to/input.jsonl`
3. **Create output dataset** — `firectl dataset create <OUTPUT_DATASET_ID>` (empty, for results)
4. **Submit batch job** — `firectl batch-inference-job create --model <MODEL> --input-dataset <INPUT_DATASET_ID> --output-dataset <OUTPUT_DATASET_ID>`
5. **Monitor progress** — `firectl batch-inference-job get <JOB_ID>` (check percent complete, token counts)
6. **Download results** — `firectl dataset download <OUTPUT_DATASET_ID> /local/path/`

## Common gotchas

- **Missing `--deployment-shape`** — Always pass `--deployment-shape default` or a specific shape when creating deployments. Shapeless deployments skip validation and are the most common cause of creation failures.
- **Region placement is immutable** — Set `--region` at deployment creation time. To change it, delete and recreate the deployment.
- **Serverless model lifecycle** — Models may be updated or deprecated with 2+ weeks notice. For long-term stability, use on-demand deployments.
- **Rate limits on serverless** — Limits are adaptive and based on spend tier. Check response headers (`X-Ratelimit-Limit-Tokens-*`) to see your current TPM ceiling.
- **Prompt caching requires session affinity** — Pass `x-session-affinity` header (or OpenAI `user` field) with a stable identifier to maximize cache hits on repeated prompts.
- **Batch API billing** — Batch jobs are billed at 50% of standard rates, but they run asynchronously (12–72 hour window). Don't use for latency-sensitive workloads.
- **LoRA deployment format** — On-demand deployments use `<MODEL>#<DEPLOYMENT_NAME>` format. Serverless uses the Specialized Model Inference API (load/unload endpoints).
- **API key expiration** — API keys expire after 1 year by default. Regenerate before expiration to avoid service interruption.
- **Deployment scaling to zero** — Deployments scale to zero after 1 hour of inactivity. Requests return 503 while scaling up; implement retry logic.
- **Training data format** — Ensure JSONL is valid (one JSON object per line). Malformed data causes validation failures before training starts.

## Verification checklist

Before submitting work with Fireworks:

- [ ] **API key is set** — `echo $FIREWORKS_API_KEY` returns a non-empty value
- [ ] **Model name is correct** — Verify format: `accounts/fireworks/models/<NAME>` (serverless) or `accounts/<ACCOUNT_ID>/deployments/<ID>` (on-demand)
- [ ] **Deployment is ready** — `firectl deployment get <ID>` shows `state: READY` and `ready_replica_count > 0`
- [ ] **Rate limits are understood** — Check response headers or account quotas for TPM/RPM limits
- [ ] **Billing model is chosen** — Confirm serverless (per-token), on-demand (per-GPU-hour), or batch (50% discount)
- [ ] **Prompt caching is configured** (if using repeated prompts) — Pass `x-session-affinity` header for cache hits
- [ ] **Error handling is in place** — Implement retries for 429 (rate limit), 503 (scaling), and transient errors
- [ ] **Costs are estimated** — Use the training cost estimator for fine-tuning jobs; calculate token costs for inference
- [ ] **Data is validated** — For training, ensure JSONL format and required fields (`messages` or `prompt`/`completion`)
- [ ] **Monitoring is set up** — Log token usage, latency, and error rates for production workloads

## Resources

**Comprehensive navigation:** https://docs.fireworks.ai/llms.txt

**Critical documentation:**
- [Serverless Quickstart](/getting-started/quickstart) — Make your first API call in minutes
- [On-demand Deployments](/guides/ondemand-deployments) — Create and manage dedicated GPU deployments
- [Fine-tuning Overview](/fine-tuning/finetuning-intro) — Train models with Managed Training or the Training API
- [API Reference](/api-reference/introduction) — Full REST API and SDK documentation
- [Batch Inference](/guides/batch-inference) — Run async jobs at 50% discount
- [Prompt Caching](/guides/prompt-caching) — Optimize repeated prompts with caching
- [Structured Outputs](/structured-responses/structured-response-formatting) — Get reliable JSON responses
- [Function Calling](/guides/function-calling) — Connect models to tools and APIs

---

> For additional documentation and navigation, see: https://docs.fireworks.ai/llms.txt