Set up Baseten with an AI agent

name: baseten
description: Use when deploying machine learning models to production, calling
  hosted model APIs, managing model inference endpoints, configuring autoscaling
  and environments, training models with Loops or Training Jobs, building
  multi-step inference pipelines with Chains, or managing model deployments
  through the CLI or Management API.
metadata:
  mintlify-proj: baseten
  version: "1.0"

Baseten Skill

Product summary

Baseten is a platform for model inference and training on production GPU infrastructure. Deploy open-source or custom models on dedicated GPUs with autoscaling, call hosted models through OpenAI-compatible APIs (Model APIs), train models with Loops (LoRA fine-tuning) or Training Jobs (custom training code), and build multi-step pipelines with Chains. Use the Baseten CLI (baseten command) to manage models, deployments, environments, and secrets. Key files: config.yaml (model configuration), model.py (custom model class), truss.yaml (Chains configuration). Primary docs: https://docs.baseten.co

When to use

Reach for this skill when:

• Deploying models: Agent needs to push a model to production, create a deployment, or manage versions
• Calling hosted models: Agent needs to use Model APIs (OpenAI Chat Completions, Anthropic Messages) without deploying
• Managing deployments: Agent needs to scale replicas, promote to production, check logs, or adjust autoscaling
• Configuring inference: Agent needs to set up streaming, structured outputs, function calling, or async inference
• Training models: Agent needs to fine-tune with Loops (LoRA + SFT/RL) or run custom training with Training Jobs
• Building pipelines: Agent needs to orchestrate multi-model workflows with Chains (separate hardware per step)
• Workspace management: Agent needs to manage API keys, secrets, environments, or billing

Quick reference

Essential CLI commands

Task	Command
Deploy a model	baseten model push
Watch for live edits (dev)	baseten model push --watch
List models	baseten model list
Get deployment logs	baseten model deployment logs <model-id>
Scale replicas	baseten model deployment update-autoscaling <model-id> --min-replica 1 --max-replica 5
Promote to production	baseten model deployment promote <model-id> <deployment-id>
Create API key	baseten org api-key create
Store secret	baseten org secret upsert <name> <value>
Authenticate	baseten auth login

config.yaml structure

model_name: my-model
resources:
  accelerator: "L4"  # or instance_type: "L4:4x16"
requirements:
  - torch==2.0.0
  - transformers==4.30.0
environment_variables:
  LOG_LEVEL: INFO

Model class basics

from truss import BaseModel

class Model(BaseModel):
    def load(self):
        # Load weights, initialize model
        pass
    
    def predict(self, model_input):
        # Return prediction
        return result

Deployment endpoints

Endpoint	Use case
/production/predict	Call production deployment (synchronous)
/development/predict	Call development deployment (testing)
/deployment/{id}/predict	Call specific deployment by ID
/production/async_predict	Queue async job to production
Model APIs base URL	https://inference.baseten.co/v1 (hosted models)

Autoscaling parameters

Parameter	Default	Purpose
min_replica	0	Minimum running replicas (0 = scale to zero)
max_replica	1	Maximum replicas (cost ceiling)
concurrency_target	1	Requests per replica before scaling up
target_utilization	70%	Headroom before scaling triggers
scale_down_delay	900s	Wait before removing idle replicas

Decision guidance

When to use X vs Y

Scenario	Use Model APIs	Use Dedicated Inference
Need a supported model now	✓
Custom model or weights		✓
Control autoscaling		✓
Pay per token	✓
Pay per GPU minute		✓
Need stable endpoint		✓

Scenario	Use Truss config.yaml	Use Python Model class
Standard LLM (Engine-Builder-LLM)	✓
Custom preprocessing/postprocessing		✓
Multiple endpoints (/chat, /embed)		✓
Streaming output	✓ or ✓	✓

Scenario	Use Loops	Use Training Jobs
LoRA fine-tuning on supported model	✓
Full-parameter training		✓
Custom training framework		✓
Supervised fine-tuning (SFT)	✓
Reinforcement learning (RL)	✓

Scenario	Use single deployment	Use Chains
Single model	✓
Multi-step pipeline (RAG, etc.)		✓
Different hardware per step		✓
Independent scaling per step		✓

Workflow

Deploy a model

1. Prepare the model: Create a directory with config.yaml and optional model.py
2. Configure resources: Set accelerator (GPU type) and requirements (dependencies) in config.yaml
3. Push to Baseten: Run baseten model push to build and deploy
4. Wait for readiness: Monitor status with baseten model deployment logs <model-id>
5. Test the endpoint: Send a prediction request to /production/predict
6. Iterate: Use baseten model push --watch for live code patching (Python models only)

Call a Model API

1. Create API key: Go to https://app.baseten.co/settings/api_keys or run baseten org api-key create
2. Set environment variable: export BASETEN_API_KEY="your_key"
3. Use OpenAI SDK: Point base URL to https://inference.baseten.co/v1 and select model slug
4. Send request: Use standard Chat Completions or Messages API format
5. Handle streaming: Use stream=True for incremental output

Promote to production

1. Validate deployment: Check logs and test with sample requests
2. Promote: Run baseten model deployment promote <model-id> <deployment-id> or use dashboard
3. Verify: Check that production environment points to new deployment
4. Monitor: Watch logs and metrics for errors or performance issues

Scale a deployment

1. Check current settings: baseten model deployment get <model-id> <deployment-id>
2. Update autoscaling: baseten model deployment update-autoscaling <model-id> --min-replica 1 --max-replica 10
3. Verify changes: Confirm settings applied with get command
4. Monitor: Watch replica count and latency during traffic changes

Train with Loops

1. Create session: baseten loops session create --model <model-name>
2. Write training script: Define sampler and trainer with Loops SDK
3. Run training: baseten loops run create --session <session-id> --config train.yaml
4. Monitor: Check logs and validation loss
5. Deploy checkpoint: baseten loops checkpoint deploy <checkpoint-id>

Common gotchas

• Cold starts: Deployments with min_replica: 0 incur startup latency on first request. Set min_replica: 1 for warm capacity.
• Async payload limit: Async predict requests have a 256 KiB size limit. Use file I/O for larger inputs.
• Development deployments are mutable: Changes apply immediately with --watch, but only Python models support live patching. Custom Docker and engine configs require rebuild.
• Promotion creates new deployment: Promoting a development deployment creates a new published deployment with a different ID. Save the returned ID.
• Secrets are team-scoped: If using teams, secrets are only accessible to models deployed to that team.
• Model APIs are read-only: You cannot deploy custom models through Model APIs; use Dedicated Inference instead.
• Streaming requires generator: Custom Truss models must return a generator (yield) to stream; synchronous returns block until complete.
• Environment variables in config.yaml are not secrets: Use baseten org secret upsert for credentials, then reference in model code via os.environ.
• Autoscaling window is lookback time: Default 60s means the autoscaler averages traffic over the last 60 seconds before scaling. Shorter windows react faster to spikes.
• Scale-down delay prevents thrashing: Default 900s (15 min) waits before removing idle replicas. Set lower for cost-sensitive workloads, higher for stable traffic.

Verification checklist

Before submitting work:

• Model builds without errors: Check baseten model deployment logs for build failures
• Endpoint is reachable: Send a test prediction to /production/predict or Model API
• Autoscaling settings are correct: Verify min_replica, max_replica, concurrency_target with get command
• Secrets are injected: Confirm model code can read environment variables set via baseten org secret upsert
• Logs are clean: No crashes, OOMKilled, or unhandled exceptions in deployment logs
• Promotion is complete: Verify environment's current_deployment.id matches promoted deployment ID
• Streaming works (if enabled): Test with stream=True and confirm chunks arrive incrementally
• API key has correct permissions: Verify key type (PERSONAL, WORKSPACE_INVOKE, etc.) matches use case
• Regional constraints are met: If using regional environments, confirm deployment region matches environment

Resources

• Comprehensive navigation: https://docs.baseten.co/llms.txt
• Deploy your first model: https://docs.baseten.co/examples/deploy-your-first-model
• Inference API reference: https://docs.baseten.co/reference/inference-api/overview
• Management API reference: https://docs.baseten.co/reference/management-api/overview
• Baseten CLI reference: https://docs.baseten.co/reference/cli/baseten/overview

---

For additional documentation and navigation, see: https://docs.baseten.co/llms.txt