Set up RunPod with an AI agent

name: runpod
description: Use when deploying GPU/CPU workloads, building serverless endpoints
  for AI inference, managing containerized applications, creating scalable APIs,
  or provisioning compute resources for training and fine-tuning. Reach for this
  skill when users need to run code on remote GPUs, deploy models, manage
  persistent storage, or automate infrastructure.
metadata:
  mintlify-proj: runpod
  version: "1.0"

RunPod Skill

Product summary

RunPod is a cloud computing platform for AI, ML, and general compute workloads. It provides three primary compute models: Pods (dedicated GPU/CPU instances), Serverless (autoscaling, pay-per-second endpoints), and Flash (Python SDK for remote function execution). Use the REST API v2 (https://api.runpod.io/v2) for resource management, the Python SDK (runpod package) for building and deploying applications, and the CLI (rp or runpodctl) for local development and operations. Key files: handler functions (Python), Dockerfiles for workers, configuration files for endpoints. Primary docs: https://docs.runpod.io

When to use

• Deploying Serverless endpoints: User wants to build autoscaling APIs for inference, batch processing, or request/response workloads. Use queue-based endpoints (/run, /runsync) or load-balanced endpoints for custom HTTP routes.
• Managing Pods: User needs dedicated GPU/CPU instances for training, fine-tuning, development, or long-running workloads. Deploy via console, CLI, or API v2.
• Building with the SDK: User wants to write Python functions that run on remote GPUs with decorators (@app.queue, @app.task, @app.api), then deploy as endpoints.
• Handling persistent data: User needs to share files across Pods or workers. Use network volumes (single datacenter) or global volumes (region-independent).
• Debugging and monitoring: User needs to check logs, SSH into workers, or inspect endpoint metrics.
• Managing credentials and secrets: User needs API keys, SSH keys, or encrypted secrets for environment variables.

Quick reference

CLI commands

Command	Purpose
rp login	Authenticate and store credentials
rp flash dev main.py	Start live dev session for local testing
rp flash deploy	Deploy apps to production endpoints
rp flash undeploy --app APP --env ENV	Remove deployed endpoint
runpodctl pod create --name POD --gpu-id GPU --image IMAGE	Create a Pod via CLI
runpodctl pod list	List all Pods
runpodctl pod stop POD_ID	Stop a Pod
runpodctl pod delete POD_ID	Terminate a Pod
runpodctl serverless create --name ENDPOINT --gpu-id GPU --image IMAGE	Create Serverless endpoint

API v2 endpoints

Resource	Method	Path
Pods	POST	/v2/pods
Pods	GET	/v2/pods
Pods	GET	/v2/pods/{id}
Pods	PATCH	/v2/pods/{id}
Pods	DELETE	/v2/pods/{id}
Serverless	POST	/v2/serverless
Serverless	GET	/v2/serverless
Serverless	GET	/v2/serverless/{id}
Network Volumes	POST	/v2/network-volumes
Network Volumes	GET	/v2/network-volumes
Secrets	POST	/v2/account/secrets
Secrets	GET	/v2/account/secrets

Handler function template

import runpod

def handler(job):
    job_input = job["input"]
    # Process input
    result = process_data(job_input)
    return result

runpod.serverless.start({"handler": handler})

Serverless request operations

Operation	Method	Purpose
/runsync	POST	Submit job, wait for result (sync)
/run	POST	Submit job, return immediately (async)
/status/{id}	GET	Check job status and results
/stream/{id}	GET	Stream incremental results
/cancel/{id}	POST	Stop a running job
/health	GET	Check endpoint health

Environment variable reference

Variable	Use	Example
RUNPOD_API_KEY	Authentication for SDK/CLI	export RUNPOD_API_KEY=...
{{ RUNPOD_SECRET_name }}	Reference encrypted secret in endpoint	env: {"TOKEN": "{{ RUNPOD_SECRET_api_key }}"}
PORT	Load balancer worker port	env: {"PORT": "8000"}
HEALTH_CHECK_PATH	Load balancer health check	env: {"HEALTH_CHECK_PATH": "/ping"}

Decision guidance

When to use Pods vs. Serverless

Scenario	Use Pods	Use Serverless
Long-running training/fine-tuning	✓	✗
Interactive development (JupyterLab, SSH)	✓	✗
Inference with variable traffic	✗	✓
Batch processing with queue	✗	✓
Predictable, continuous load	✓	✗
Pay-per-second, scale to zero	✗	✓
Custom HTTP API routes	✗	✓ (load balancer)

When to use network volume vs. global volume

Scenario	Network Volume	Global Volume
Single datacenter workload	✓	✗
Multi-region access	✗	✓
Fast writes/POSIX compliance	✓	✗
Model weights shared across workers	✓	✓
Tied to specific data center	✓	✗
Region-independent storage	✗	✓

When to use queue-based vs. load-balanced endpoints

Scenario	Queue-based	Load-balanced
Handler function pattern	✓	✗
Custom HTTP routes (FastAPI, Flask)	✗	✓
Async job processing	✓	✗
Real-time HTTP API	✗	✓
Job queuing and status tracking	✓	✗
WebSocket support	✗	✓

Workflow

Deploy a Serverless endpoint (queue-based)

1. Write handler function: Create handler.py with a function that accepts job dict, extracts job["input"], processes it, and returns result.
2. Test locally: Run python handler.py or use rp flash dev to test with live code reloading.
3. Create Dockerfile: Package handler, dependencies, and base image. Use FROM runpod/base:latest or similar.
4. Push to registry: Build and push image to Docker Hub or GitHub Container Registry.
5. Deploy endpoint: Via console (Serverless → New Endpoint → Docker) or CLI: runpodctl serverless create --name my-endpoint --gpu-id "NVIDIA A40" --image my-image:latest.
6. Configure scaling: Set min/max workers, idle timeout, and scaling policy (queue delay or request count).
7. Send requests: Use /runsync for sync jobs or /run for async, with {"input": {...}} payload.
8. Monitor: Check logs, worker status, and metrics in console or via API.

Deploy a Pod for development

1. Choose GPU: Decide on GPU type, region, and cloud tier (Secure or Community).
2. Select template: Pick a pre-configured template (PyTorch, TensorFlow, etc.) or custom image.
3. Configure storage: Attach network volume for persistent data or use volume disk.
4. Deploy: Via console (Pods → Deploy) or CLI: runpodctl pod create --name dev-pod --gpu-id "NVIDIA RTX 4090" --image runpod/pytorch:latest.
5. Connect: SSH, JupyterLab, or VS Code via console or CLI.
6. Develop: Install packages, run code, save work to network volume.
7. Stop/terminate: Stop to pause billing (storage still charged), or terminate to delete.

Build and deploy with the SDK (Flash)

1. Install SDK: pip install runpod (requires Python 3.10+).
2. Authenticate: rp login to store credentials.
3. Create app: Define runpod.App("name") with decorated functions (@app.queue, @app.task, @app.api).
4. Add local entrypoint: Mark main function with @runpod.local_entrypoint.
5. Test locally: rp flash dev main.py to run functions on remote GPUs from your machine.
6. Deploy: rp flash deploy to create production endpoints.
7. Call functions: Use .remote(), .spawn(), or .stream() to invoke remote functions.

Manage secrets and environment variables

1. Create secret: Via console (Settings → Secrets) or API: POST /v2/account/secrets with name and value.
2. Reference in endpoint: Set env var to {{ RUNPOD_SECRET_name }} in endpoint config.
3. Access in code: Read from os.environ["VAR_NAME"] in handler or Pod.
4. Rotate: Update secret value via API or console; new workers pick up new value.

Common gotchas

• Cold starts: Serverless workers take time to initialize (load models, install packages). Minimize by using cached models, enabling FlashBoot, or keeping min workers > 0.
• Payload size limits: /run max 10 MB, /runsync max 20 MB. For larger results, store in S3 or network volume and return links.
• Handler initialization: Load models and heavy resources outside the handler function, not inside. Inside handler runs on every request.
• GPU availability: Requested GPU may be out of capacity. Use API to check availability before creating Pods, or subscribe to "deploy when available."
• Network volume datacenter: Network volumes are tied to one datacenter. Pod must be in same datacenter to attach. Global volumes have no datacenter constraint.
• Secrets are write-only: Secret values are never returned by API. Store securely and rotate periodically.
• Worker refresh: Long-running workers accumulate state. Use "refresh_worker": True in handler return to clear state after job.
• Timeout confusion: executionTimeout is max runtime once worker picks up job. ttl is total lifespan from submission to deletion. Both have 7-day max.
• Rate limits scale with workers: Base limit + (running_workers × per-worker_limit). Hitting 429? Implement backoff and retry logic.
• Docker Compose not supported: Runpod runs Docker for you. Cannot use Docker Compose or docker-in-docker on Pods.
• UDP not supported: Only TCP and HTTP connections. No UDP sockets.
• Async handler cleanup: Catch asyncio.CancelledError to release resources when job is cancelled, then re-raise.

Verification checklist

Before submitting work:

• Handler function tested locally with python handler.py or rp flash dev.
• Dockerfile builds without errors and includes all dependencies.
• Environment variables (including secrets) are set correctly in endpoint config.
• API key has appropriate permissions (check in console under Settings → API Keys).
• Network volume (if used) is in same datacenter as Pod/endpoint.
• Request payload is under size limits (10 MB for /run, 20 MB for /runsync).
• Endpoint scaling settings (min/max workers, idle timeout) match workload.
• Logs are checked for errors: console Logs tab or rp logs POD_ID --follow.
• Cold start time is acceptable (test with /runsync first request).
• Results are retrieved within retention window (30 min async, 1 min sync).

Resources

• Comprehensive navigation: https://docs.runpod.io/llms.txt
• API v2 reference: https://docs.runpod.io/api-reference-v2/overview
• Serverless quickstart: https://docs.runpod.io/serverless/quickstart
• SDK overview and examples: https://docs.runpod.io/sdk/overview

---

For additional documentation and navigation, see: https://docs.runpod.io/llms.txt