Set up Braintrust with an AI agent

name: braintrust
description: Use when building, evaluating, and improving AI applications. Reach
  for this skill when you need to instrument applications with tracing, run
  systematic evaluations, analyze production logs, build datasets, or deploy AI
  systems with observability and version control.
metadata:
  mintlify-proj: braintrust
  version: "1.0"

Braintrust Skill

Product summary

Braintrust is an active observability platform for AI applications that captures traces from LLM calls and application logic, analyzes production behavior, runs systematic evaluations, and deploys changes with version control. Agents use Braintrust to instrument applications (via SDKs in TypeScript, Python, Go, Java, Ruby, C#), run experiments to compare prompts and models, build evaluation datasets from production logs, and monitor deployed systems.

Key files and commands:

• .bt/config.json — Project configuration (created by bt init)
• BRAINTRUST_API_KEY — Environment variable for authentication
• bt CLI — Command-line interface for tracing, evals, querying, and syncing
• SDKs: braintrust (TypeScript/Python), github.com/braintrust-dev/braintrust-go (Go), etc.
• Primary docs: https://docs.braintrust.dev

When to use

Reach for this skill when:

• Instrumenting: Adding tracing to capture LLM calls, application logic, and user feedback from AI applications
• Observing: Analyzing production logs, filtering traces, discovering patterns, or investigating issues
• Evaluating: Running experiments to compare prompts, models, or agent configurations; building datasets; writing scorers
• Deploying: Versioning and deploying prompts, functions, or using the Braintrust Gateway for multi-provider routing
• Annotating: Adding human feedback, labels, or corrections to traces; building evaluation datasets from production
• Automating: Setting up online scoring rules, automations, or CI/CD integration for regression detection

Do not use for: Account creation, billing, or SSO setup (admin-only tasks).

Quick reference

Authentication and setup

Task	Command/Code
Create API key	Go to Settings > API keys in Braintrust UI
Set API key	export BRAINTRUST_API_KEY="your-key"
Initialize project	bt init (creates .bt/config.json)
Switch project/org	bt switch
Check current context	bt status

Core SDK patterns

Pattern	TypeScript	Python
Initialize logger	initLogger({ projectName: "..." })	braintrust.init_logger(project="...")
Log a trace	logger.log({ input, output })	logger.log(input=..., output=...)
Create experiment	init({ project: "...", experiment: "..." })	braintrust.init(project="...", experiment="...")
Create dataset	initDataset("project", { dataset: "name" })	braintrust.init_dataset(project="...", name="...")
Run evaluation	Eval({ data, task, scores })	braintrust.Eval(data=..., task=..., scores=...)
Flush pending spans	await logger.flush()	logger.flush()

CLI commands

Command	Purpose
bt trace enable <agent>	Enable tracing for a coding agent (Claude, Cursor, etc.)
bt eval <file>	Run evaluation from JS/TS/Python/Go file
bt sql "SELECT ..."	Query logs with SQL
bt view logs	Browse logs in terminal UI
bt sync pull	Download logs/experiments as NDJSON
bt sync push	Upload local NDJSON to Braintrust
bt prompts list	List prompts in current project
bt datasets list	List datasets
bt experiments list	List experiments

Environment variables

Variable	Purpose
BRAINTRUST_API_KEY	API key for authentication (required)
BRAINTRUST_DEFAULT_PROJECT	Default project name for spans
BRAINTRUST_ORG_NAME	Organization name (when accessing multiple orgs)
BRAINTRUST_DEBUG	Set to true for debug logging
BRAINTRUST_ENABLE_TRACE_CONSOLE_LOG	Set to true to print spans to console

Decision guidance

When to use logger vs. experiment

Use case	Logger	Experiment
Capture production traces	✓	—
Run one-off evaluations	—	✓
Build datasets from logs	✓	—
Compare multiple runs	—	✓
Mutable/iterative results	✓	—
Immutable snapshots	—	✓

When to use playground vs. experiment vs. online scoring

Approach	Use for	Mutability
Playground	Rapid iteration, prototyping	Mutable (overwrites)
Experiment	Immutable snapshots, CI/CD, tracking progress	Immutable
Online scoring	Production monitoring, continuous evaluation	Async, real-time

When to use code scorer vs. LLM-as-a-judge vs. autoeval

Scorer type	Best for	Pros	Cons
Code scorer	Deterministic checks (exact match, format)	Fast, cheap, precise	Requires logic
LLM-as-a-judge	Subjective quality (helpfulness, tone)	Flexible, handles nuance	Slower, more expensive
Autoeval	Common patterns (similarity, relevance)	Pre-built, no code	Limited to built-in metrics

Workflow

Typical task: Instrument, evaluate, and improve an AI application

1. Set up authentication

   • Create API key in Braintrust UI (Settings > API keys)
   • Set BRAINTRUST_API_KEY environment variable
   • Run bt init to link your project directory

2. Instrument your application

   • Install SDK: npm install braintrust (TypeScript) or pip install braintrust (Python)
   • Call initLogger() at application startup
   • Wrap LLM calls with auto-instrumentation or manual logger.log()
   • Deploy to production; traces flow to Braintrust automatically

3. Observe production behavior

   • Go to Logs page in Braintrust UI
   • Filter and search traces using SQL or UI filters
   • Use Loop to ask natural language questions about your data
   • Enable Patterns to surface recurring issues automatically

4. Build evaluation datasets

   • Identify interesting traces in production logs
   • Add human feedback (scores, corrections, labels)
   • Promote annotated traces to a dataset
   • Alternatively, create dataset from scratch with initDataset()

5. Write scorers

   • For deterministic checks: write code-based scorer
   • For subjective quality: use LLM-as-a-judge scorer
   • Test scorer on obvious cases (correct output should score ~1, wrong ~0)

6. Run experiments

   • Create experiment with init({ experiment: "name" })
   • Define task function (what to evaluate)
   • Pass dataset and scorers to Eval()
   • Compare results against baseline experiment

7. Deploy improvements

   • Promote winning prompt/model to production
   • Use logger.invoke() to call deployed prompts
   • Set up online scoring rules to monitor production quality
   • Create dashboards to track metrics over time

Typical task: Run an evaluation from code

import { Eval, initDataset } from "braintrust";

const dataset = initDataset("My Project", { dataset: "test-cases" });

await Eval("My Project", {
  data: dataset,
  task: async (input) => {
    // Your task: call model, agent, or function
    return await myModel.call(input);
  },
  scores: [
    {
      name: "correctness",
      score: async (input, output, expected) => {
        // Return 0-1 score
        return output === expected ? 1 : 0;
      },
    },
  ],
  maxConcurrency: 10, // Limit concurrency to avoid rate limits
});

Common gotchas

• Missing API key: Always set BRAINTRUST_API_KEY before running code. Check with echo $BRAINTRUST_API_KEY.
• Forgetting to flush: Call logger.flush() before process exit to ensure pending spans are sent. Use BRAINTRUST_FLUSH_ON_EXIT=true (default) to auto-flush.
• Mixing logger and experiment spans: Don't use initLogger() and init() in the same trace hierarchy. Use separate instances.
• Unlimited concurrency in evals: Set maxConcurrency to avoid "too many open files" errors. Start with 10, tune up/down.
• Scorer not testing obvious cases: Always test scorers on inputs where the correct score is clear (1 for correct, 0 for wrong). If it fails, fix the scorer before trusting results.
• Experiment name collisions: Experiment names must be unique within a project. Use timestamps or version numbers to avoid conflicts.
• Dataset search limited to 1000 records: Use "Add to dataset" search in UI for larger datasets, or query via API.
• Traces stuck in progress: Ensure spans are flushed and process exits cleanly. Check logs for errors.
• Custom code scorers can't import packages: Inline all dependencies or use LLM-as-a-judge instead.
• OTEL payload limits: Large spans may be rejected. Keep individual span payloads under size limits; split if needed.

Verification checklist

Before submitting work with Braintrust:

• API key is set and valid (bt status shows correct org/project)
• Traces are being logged (check Logs page in UI or bt view logs)
• Spans are flushed before process exit (logger.flush() called or BRAINTRUST_FLUSH_ON_EXIT=true)
• Experiment names are unique within the project
• Scorers tested on obvious cases (correct output scores ~1, wrong ~0)
• Dataset has sufficient examples (10-200 for most use cases)
• Concurrency is limited in evals (maxConcurrency set to avoid resource exhaustion)
• Comparison baseline is set when comparing experiments
• Online scoring rules have appropriate filters and sample rates
• Dashboards and alerts are configured for production monitoring

Resources

• Comprehensive navigation: https://docs.braintrust.dev/llms.txt
• Getting started: https://docs.braintrust.dev (main docs site)
• Workflow overview: https://docs.braintrust.dev/workflow
• Instrument your app: https://docs.braintrust.dev/instrument
• Run evaluations: https://docs.braintrust.dev/evaluate/run-evaluations
• CLI reference: https://docs.braintrust.dev/reference/cli/overview
• SDK reference: https://docs.braintrust.dev/sdks (TypeScript, Python, Go, Java, Ruby, C#)
• API reference: https://docs.braintrust.dev/api-reference

---

For additional documentation and navigation, see: https://docs.braintrust.dev/llms.txt