Set up Together AI with an AI agent

name: together
description: Use when running inference on open-source LLMs, fine-tuning models,
  deploying dedicated endpoints, building agents with function calling, or
  managing GPU clusters. Agents should reach for this skill when users need to
  call models via API, customize models with training data, set up production
  inference infrastructure, or build AI applications with tool use.
metadata:
  mintlify-proj: together
  version: "1.0"

Together AI Skill

Product summary

Together AI is a platform for running, training, and serving open-source AI models. Agents use it to call 100+ models via serverless inference (pay-per-token), deploy dedicated endpoints (pay-per-GPU-minute), fine-tune models on custom data, or manage GPU clusters for large-scale workloads.

Key files and commands:

• API endpoint: https://api.together.ai/v1/chat/completions (serverless) or https://api-inference.together.ai (dedicated)
• CLI: tg (Together CLI 3.0+) — authenticate with tg login, deploy with tg endpoints deploy, manage training with tg fine-tuning
• SDKs: Python (together) and TypeScript (together-ai) — both read TOGETHER_API_KEY from environment
• Config: API keys are project-scoped; create at https://api.together.ai/settings/projects/~current/api-keys
• Primary docs: https://docs.together.ai

When to use

Reach for this skill when:

• Running inference: User asks to call a model (chat, vision, image generation, video, audio, reasoning)
• Choosing deployment: User needs serverless (variable traffic, prototyping), provisioned throughput (production SLA), or dedicated endpoints (fine-tuned models, steady traffic, custom hardware)
• Fine-tuning: User has training data and wants to adapt a model with SFT, DPO, or full fine-tuning
• Building agents: User needs function calling, multi-turn loops, or tool use workflows
• Production setup: User needs monitoring, traffic splitting, A/B tests, rollouts, or autoscaling
• GPU clusters: User needs to run large training jobs or batch workloads on reserved hardware

Quick reference

Inference modes

Mode	Billing	Best for	Setup
Serverless	Per token	Prototyping, variable traffic	No setup; call model="moonshotai/Kimi-K3"
Provisioned throughput	Per PTU (fixed term)	Production, defined SLA	Contact sales
Dedicated endpoints	Per GPU-minute	Fine-tuned models, steady traffic, custom hardware	tg endpoints deploy <model> --endpoint <name>

Common models

Use case	Recommended model	API string
Fast chat	MiniMax M3.1 Flash	MiniMaxAI/MiniMax-M3.1-Flash
Reasoning	DeepSeek V4 Pro	deepseek-ai/DeepSeek-V4-Pro-0813
Vision	Qwen3.5 9B	Qwen/Qwen3.5-9B
Image generation	FLUX.2 dev	black-forest-labs/FLUX.2-dev
Video generation	Seedance 2.5	ByteDance/Seedance-2.5
Speech-to-text	Whisper large-v3	openai/whisper-large-v3

CLI essentials

# Authentication
tg login                              # Interactive login
tg logout                             # Clear credentials
export TOGETHER_API_KEY="..."         # Use API key instead

# Inference
tg models list                        # Browse serverless models
tg models configs <model_id>          # Show hardware configs for dedicated

# Endpoints (dedicated inference)
tg endpoints deploy <model> --endpoint <name>    # Create and deploy
tg endpoints ls                       # List endpoints
tg endpoints get <endpoint_id>        # Check status
tg endpoints update <deployment_id> --min-replicas 0 --max-replicas 0  # Stop
tg endpoints rm <endpoint_id> --force # Delete

# Fine-tuning
tg files upload <file.jsonl>          # Upload training data
tg fine-tuning create --training-file <FILE_ID> --model <base_model> --lora  # Start job
tg fine-tuning retrieve <JOB_ID>      # Check progress
tg fine-tuning list                   # List all jobs

# Training (post-training with custom loop)
tg training list                      # List resources
tg training model-limits <model>      # Check constraints

SDK patterns

Python:

from together import Together
client = Together()  # reads TOGETHER_API_KEY

# Serverless inference
response = client.chat.completions.create(
    model="MiniMaxAI/MiniMax-M3.1-Flash",
    messages=[{"role": "user", "content": "Hello"}],
    stream=False,
)

# Dedicated endpoint
response = client.chat.completions.create(
    model="your-project-slug/my-endpoint",
    messages=[{"role": "user", "content": "Hello"}],
)

# With function calling
response = client.chat.completions.create(
    model="MiniMaxAI/MiniMax-M3.1-Flash",
    messages=[{"role": "user", "content": "What's the weather?"}],
    tools=[{"type": "function", "function": {...}}],
)

TypeScript:

import Together from 'together-ai';
const client = new Together();

const response = await client.chat.completions.create({
  model: 'MiniMaxAI/MiniMax-M3.1-Flash',
  messages: [{ role: 'user', content: 'Hello' }],
});

Decision guidance

Serverless vs. dedicated endpoints

Condition	Use serverless	Use dedicated
Traffic is variable or bursty	✓
Traffic is steady and predictable		✓
Model is fine-tuned		✓
Need custom hardware (H100, B200)		✓
Need latency SLA		✓
Prototyping or testing	✓
High volume (>1M tokens/day)		✓

LoRA vs. full fine-tuning

Factor	LoRA	Full
Cost	Lower	Higher
Speed	Faster	Slower
Quality on in-domain data	Good	Better
Quality on out-of-domain data	May degrade	Maintains better
Deployment size	Smaller	Larger
When to use	Most cases	Major domain shift

Training methods

Method	Use when	Billing
Fine-tuning (SFT)	You have labeled examples of desired behavior	Per token processed
Preference tuning (DPO)	You have preferred/dispreferred response pairs	Per token processed
Post-training (SFT/DPO/GRPO)	You need step-level control, custom loss, or RL	Per GPU-minute

Workflow

1. Make your first inference request

1. Create an API key at https://api.together.ai/settings/projects/~current/api-keys
2. Set export TOGETHER_API_KEY="your_key"
3. Install SDK: pip install together (Python) or npm install together-ai (TypeScript)
4. Run a request:

   from together import Together
   client = Together()
   response = client.chat.completions.create(
       model="MiniMaxAI/MiniMax-M3.1-Flash",
       messages=[{"role": "user", "content": "What is 2+2?"}],
   )
   print(response.choices[0].message.content)
   

5. Verify the response contains choices[0].message.content

2. Deploy a dedicated endpoint

1. Choose a model from /docs/dedicated-endpoints/models
2. Run: tg endpoints deploy <model_id> --endpoint <name>
3. Wait for status to show DEPLOYMENT_STATE_READY: tg endpoints get <endpoint_id>
4. Call it like serverless, but use model="<project_slug>/<endpoint_name>"
5. Stop when done: tg endpoints update <deployment_id> --min-replicas 0 --max-replicas 0
6. Delete: tg endpoints rm <endpoint_id> --force

3. Fine-tune a model

1. Prepare training data in JSONL format (see /docs/fine-tuning/data-preparation)
2. Upload: tg files upload train.jsonl
3. Start job: tg fine-tuning create --training-file <FILE_ID> --model <base_model> --lora
4. Monitor: tg fine-tuning retrieve <JOB_ID> (check status field)
5. Deploy: Once complete, tg endpoints deploy <fine_tuned_model_id> --endpoint <name>
6. Call the endpoint like any other model

4. Build an agent with function calling

1. Define tools as JSON schema objects with type: "function" and function: {name, description, parameters}
2. Send request with tools parameter
3. Check response for tool_calls in choices[0].message
4. Execute each tool call in your code
5. Append tool results to messages as role: "tool" with tool_call_id
6. Send messages back to model (loop until no more tool_calls)
7. Cap loops at 20–50 iterations to prevent spirals

5. Stream responses

1. Set stream=True (Python) or stream: true (TypeScript)
2. Iterate over chunks; each has choices[0].delta
3. For reasoning models, check delta.reasoning_content or delta.reasoning first, then delta.content
4. Accumulate deltas until finish_reason is not null

Common gotchas

• API key not found: Ensure TOGETHER_API_KEY is set in environment or passed to Together(api_key=...)
• Model not available: Serverless and dedicated endpoints have different model catalogs; check /docs/serverless/models vs. /docs/dedicated-endpoints/models
• Endpoint still provisioning: New deployments take 1–5 minutes to reach DEPLOYMENT_STATE_READY; poll with tg endpoints get before sending requests
• Rate limit (429): Reduce request rate, use exponential backoff, or upgrade to dedicated endpoints for higher throughput
• Context length exceeded: Set context_length_exceeded_behavior="truncate" to auto-truncate, or lower max_tokens
• Function calling not working: Verify model supports it (check /docs/serverless/models for "Function calling" column); not all models support tool use
• Fine-tuning data format: JSONL must match schema exactly; validate with tg fine-tuning preview --training-file <FILE_ID> --model <base_model>
• Streaming with function calls: Streaming works but tool calls arrive in chunks; accumulate delta.tool_calls until finish_reason is set
• Dedicated endpoint costs: Billing starts immediately when deployment reaches READY, even if idle; scale to 0 replicas to pause
• Project scope: CLI and SDK operations are scoped to a project; set TOGETHER_PROJECT_ID or pass --project to CLI commands

Verification checklist

Before submitting work:

• API key is set and valid (test with tg whoami)
• Model name is correct for the deployment mode (serverless vs. dedicated)
• For dedicated endpoints: deployment status is DEPLOYMENT_STATE_READY before sending requests
• For fine-tuning: training data is valid JSONL/Parquet and matches the schema
• For function calling: tools are defined with required fields (name, description, parameters)
• Streaming is enabled if low latency is needed
• Error handling includes retry logic for 429, 503, 504 responses
• For production: monitoring is set up (check /docs/dedicated-endpoints/monitoring)
• Project ID is correct if using multiple projects

Resources

• Comprehensive navigation: https://docs.together.ai/llms.txt
• Quickstart: https://docs.together.ai/docs/quickstart
• Serverless models catalog: https://docs.together.ai/docs/serverless/models
• Dedicated endpoints: https://docs.together.ai/docs/dedicated-endpoints/overview
• Function calling patterns: https://docs.together.ai/docs/inference/function-calling/overview
• Fine-tuning guide: https://docs.together.ai/docs/fine-tuning/overview
• Error codes: https://docs.together.ai/docs/error-codes
• CLI reference: https://docs.together.ai/reference/cli/getting-started

---

For additional documentation and navigation, see: https://docs.together.ai/llms.txt