---
name: baseten
description: Use when deploying machine learning models to production, calling
  hosted model APIs, managing model inference endpoints, configuring autoscaling
  and environments, training models with Loops or Training Jobs, building
  multi-step inference pipelines with Chains, or managing model deployments
  through the CLI or Management API.
metadata:
  mintlify-proj: baseten
  version: "1.0"
---

# Baseten Skill

## Product summary

Baseten is a platform for model inference and training on production GPU infrastructure. Deploy open-source or custom models on dedicated GPUs with autoscaling, call hosted models through OpenAI-compatible APIs (Model APIs), train models with Loops (LoRA fine-tuning) or Training Jobs (custom training code), and build multi-step pipelines with Chains. Use the Baseten CLI (`baseten` command) to manage models, deployments, environments, and secrets. Key files: `config.yaml` (model configuration), `model.py` (custom model class), `truss.yaml` (Chains configuration). Primary docs: https://docs.baseten.co

## When to use

Reach for this skill when:
- **Deploying models**: Agent needs to push a model to production, create a deployment, or manage versions
- **Calling hosted models**: Agent needs to use Model APIs (OpenAI Chat Completions, Anthropic Messages) without deploying
- **Managing deployments**: Agent needs to scale replicas, promote to production, check logs, or adjust autoscaling
- **Configuring inference**: Agent needs to set up streaming, structured outputs, function calling, or async inference
- **Training models**: Agent needs to fine-tune with Loops (LoRA + SFT/RL) or run custom training with Training Jobs
- **Building pipelines**: Agent needs to orchestrate multi-model workflows with Chains (separate hardware per step)
- **Workspace management**: Agent needs to manage API keys, secrets, environments, or billing

## Quick reference

### Essential CLI commands

| Task | Command |
|------|---------|
| Deploy a model | `baseten model push` |
| Watch for live edits (dev) | `baseten model push --watch` |
| List models | `baseten model list` |
| Get deployment logs | `baseten model deployment logs <model-id>` |
| Scale replicas | `baseten model deployment update-autoscaling <model-id> --min-replica 1 --max-replica 5` |
| Promote to production | `baseten model deployment promote <model-id> <deployment-id>` |
| Create API key | `baseten org api-key create` |
| Store secret | `baseten org secret upsert <name> <value>` |
| Authenticate | `baseten auth login` |

### config.yaml structure

```yaml
model_name: my-model
resources:
  accelerator: "L4"  # or instance_type: "L4:4x16"
requirements:
  - torch==2.0.0
  - transformers==4.30.0
environment_variables:
  LOG_LEVEL: INFO
```

### Model class basics

```python
from truss import BaseModel

class Model(BaseModel):
    def load(self):
        # Load weights, initialize model
        pass
    
    def predict(self, model_input):
        # Return prediction
        return result
```

### Deployment endpoints

| Endpoint | Use case |
|----------|----------|
| `/production/predict` | Call production deployment (synchronous) |
| `/development/predict` | Call development deployment (testing) |
| `/deployment/{id}/predict` | Call specific deployment by ID |
| `/production/async_predict` | Queue async job to production |
| Model APIs base URL | `https://inference.baseten.co/v1` (hosted models) |

### Autoscaling parameters

| Parameter | Default | Purpose |
|-----------|---------|---------|
| `min_replica` | 0 | Minimum running replicas (0 = scale to zero) |
| `max_replica` | 1 | Maximum replicas (cost ceiling) |
| `concurrency_target` | 1 | Requests per replica before scaling up |
| `target_utilization` | 70% | Headroom before scaling triggers |
| `scale_down_delay` | 900s | Wait before removing idle replicas |

## Decision guidance

### When to use X vs Y

| Scenario | Use Model APIs | Use Dedicated Inference |
|----------|---|---|
| Need a supported model now | ✓ | |
| Custom model or weights | | ✓ |
| Control autoscaling | | ✓ |
| Pay per token | ✓ | |
| Pay per GPU minute | | ✓ |
| Need stable endpoint | | ✓ |

| Scenario | Use Truss config.yaml | Use Python Model class |
|----------|---|---|
| Standard LLM (Engine-Builder-LLM) | ✓ | |
| Custom preprocessing/postprocessing | | ✓ |
| Multiple endpoints (/chat, /embed) | | ✓ |
| Streaming output | ✓ or ✓ | ✓ |

| Scenario | Use Loops | Use Training Jobs |
|----------|---|---|
| LoRA fine-tuning on supported model | ✓ | |
| Full-parameter training | | ✓ |
| Custom training framework | | ✓ |
| Supervised fine-tuning (SFT) | ✓ | |
| Reinforcement learning (RL) | ✓ | |

| Scenario | Use single deployment | Use Chains |
|----------|---|---|
| Single model | ✓ | |
| Multi-step pipeline (RAG, etc.) | | ✓ |
| Different hardware per step | | ✓ |
| Independent scaling per step | | ✓ |

## Workflow

### Deploy a model

1. **Prepare the model**: Create a directory with `config.yaml` and optional `model.py`
2. **Configure resources**: Set `accelerator` (GPU type) and `requirements` (dependencies) in `config.yaml`
3. **Push to Baseten**: Run `baseten model push` to build and deploy
4. **Wait for readiness**: Monitor status with `baseten model deployment logs <model-id>`
5. **Test the endpoint**: Send a prediction request to `/production/predict`
6. **Iterate**: Use `baseten model push --watch` for live code patching (Python models only)

### Call a Model API

1. **Create API key**: Go to https://app.baseten.co/settings/api_keys or run `baseten org api-key create`
2. **Set environment variable**: `export BASETEN_API_KEY="your_key"`
3. **Use OpenAI SDK**: Point base URL to `https://inference.baseten.co/v1` and select model slug
4. **Send request**: Use standard Chat Completions or Messages API format
5. **Handle streaming**: Use `stream=True` for incremental output

### Promote to production

1. **Validate deployment**: Check logs and test with sample requests
2. **Promote**: Run `baseten model deployment promote <model-id> <deployment-id>` or use dashboard
3. **Verify**: Check that production environment points to new deployment
4. **Monitor**: Watch logs and metrics for errors or performance issues

### Scale a deployment

1. **Check current settings**: `baseten model deployment get <model-id> <deployment-id>`
2. **Update autoscaling**: `baseten model deployment update-autoscaling <model-id> --min-replica 1 --max-replica 10`
3. **Verify changes**: Confirm settings applied with `get` command
4. **Monitor**: Watch replica count and latency during traffic changes

### Train with Loops

1. **Create session**: `baseten loops session create --model <model-name>`
2. **Write training script**: Define sampler and trainer with Loops SDK
3. **Run training**: `baseten loops run create --session <session-id> --config train.yaml`
4. **Monitor**: Check logs and validation loss
5. **Deploy checkpoint**: `baseten loops checkpoint deploy <checkpoint-id>`

## Common gotchas

- **Cold starts**: Deployments with `min_replica: 0` incur startup latency on first request. Set `min_replica: 1` for warm capacity.
- **Async payload limit**: Async predict requests have a 256 KiB size limit. Use file I/O for larger inputs.
- **Development deployments are mutable**: Changes apply immediately with `--watch`, but only Python models support live patching. Custom Docker and engine configs require rebuild.
- **Promotion creates new deployment**: Promoting a development deployment creates a new published deployment with a different ID. Save the returned ID.
- **Secrets are team-scoped**: If using teams, secrets are only accessible to models deployed to that team.
- **Model APIs are read-only**: You cannot deploy custom models through Model APIs; use Dedicated Inference instead.
- **Streaming requires generator**: Custom Truss models must return a generator (yield) to stream; synchronous returns block until complete.
- **Environment variables in config.yaml are not secrets**: Use `baseten org secret upsert` for credentials, then reference in model code via `os.environ`.
- **Autoscaling window is lookback time**: Default 60s means the autoscaler averages traffic over the last 60 seconds before scaling. Shorter windows react faster to spikes.
- **Scale-down delay prevents thrashing**: Default 900s (15 min) waits before removing idle replicas. Set lower for cost-sensitive workloads, higher for stable traffic.

## Verification checklist

Before submitting work:

- [ ] Model builds without errors: Check `baseten model deployment logs` for build failures
- [ ] Endpoint is reachable: Send a test prediction to `/production/predict` or Model API
- [ ] Autoscaling settings are correct: Verify `min_replica`, `max_replica`, `concurrency_target` with `get` command
- [ ] Secrets are injected: Confirm model code can read environment variables set via `baseten org secret upsert`
- [ ] Logs are clean: No crashes, OOMKilled, or unhandled exceptions in deployment logs
- [ ] Promotion is complete: Verify environment's `current_deployment.id` matches promoted deployment ID
- [ ] Streaming works (if enabled): Test with `stream=True` and confirm chunks arrive incrementally
- [ ] API key has correct permissions: Verify key type (PERSONAL, WORKSPACE_INVOKE, etc.) matches use case
- [ ] Regional constraints are met: If using regional environments, confirm deployment region matches environment

## Resources

- **Comprehensive navigation**: https://docs.baseten.co/llms.txt
- **Deploy your first model**: https://docs.baseten.co/examples/deploy-your-first-model
- **Inference API reference**: https://docs.baseten.co/reference/inference-api/overview
- **Management API reference**: https://docs.baseten.co/reference/management-api/overview
- **Baseten CLI reference**: https://docs.baseten.co/reference/cli/baseten/overview

---

> For additional documentation and navigation, see: https://docs.baseten.co/llms.txt