---
name: runpod
description: Use when deploying GPU/CPU workloads, building serverless endpoints
  for AI inference, managing containerized applications, creating scalable APIs,
  or provisioning compute resources for training and fine-tuning. Reach for this
  skill when users need to run code on remote GPUs, deploy models, manage
  persistent storage, or automate infrastructure.
metadata:
  mintlify-proj: runpod
  version: "1.0"
---

# RunPod Skill

## Product summary

RunPod is a cloud computing platform for AI, ML, and general compute workloads. It provides three primary compute models: **Pods** (dedicated GPU/CPU instances), **Serverless** (autoscaling, pay-per-second endpoints), and **Flash** (Python SDK for remote function execution). Use the REST API v2 (`https://api.runpod.io/v2`) for resource management, the Python SDK (`runpod` package) for building and deploying applications, and the CLI (`rp` or `runpodctl`) for local development and operations. Key files: handler functions (Python), Dockerfiles for workers, configuration files for endpoints. Primary docs: https://docs.runpod.io

## When to use

- **Deploying Serverless endpoints**: User wants to build autoscaling APIs for inference, batch processing, or request/response workloads. Use queue-based endpoints (`/run`, `/runsync`) or load-balanced endpoints for custom HTTP routes.
- **Managing Pods**: User needs dedicated GPU/CPU instances for training, fine-tuning, development, or long-running workloads. Deploy via console, CLI, or API v2.
- **Building with the SDK**: User wants to write Python functions that run on remote GPUs with decorators (`@app.queue`, `@app.task`, `@app.api`), then deploy as endpoints.
- **Handling persistent data**: User needs to share files across Pods or workers. Use network volumes (single datacenter) or global volumes (region-independent).
- **Debugging and monitoring**: User needs to check logs, SSH into workers, or inspect endpoint metrics.
- **Managing credentials and secrets**: User needs API keys, SSH keys, or encrypted secrets for environment variables.

## Quick reference

### CLI commands

| Command | Purpose |
|---------|---------|
| `rp login` | Authenticate and store credentials |
| `rp flash dev main.py` | Start live dev session for local testing |
| `rp flash deploy` | Deploy apps to production endpoints |
| `rp flash undeploy --app APP --env ENV` | Remove deployed endpoint |
| `runpodctl pod create --name POD --gpu-id GPU --image IMAGE` | Create a Pod via CLI |
| `runpodctl pod list` | List all Pods |
| `runpodctl pod stop POD_ID` | Stop a Pod |
| `runpodctl pod delete POD_ID` | Terminate a Pod |
| `runpodctl serverless create --name ENDPOINT --gpu-id GPU --image IMAGE` | Create Serverless endpoint |

### API v2 endpoints

| Resource | Method | Path |
|----------|--------|------|
| Pods | POST | `/v2/pods` |
| Pods | GET | `/v2/pods` |
| Pods | GET | `/v2/pods/{id}` |
| Pods | PATCH | `/v2/pods/{id}` |
| Pods | DELETE | `/v2/pods/{id}` |
| Serverless | POST | `/v2/serverless` |
| Serverless | GET | `/v2/serverless` |
| Serverless | GET | `/v2/serverless/{id}` |
| Network Volumes | POST | `/v2/network-volumes` |
| Network Volumes | GET | `/v2/network-volumes` |
| Secrets | POST | `/v2/account/secrets` |
| Secrets | GET | `/v2/account/secrets` |

### Handler function template

```python
import runpod

def handler(job):
    job_input = job["input"]
    # Process input
    result = process_data(job_input)
    return result

runpod.serverless.start({"handler": handler})
```

### Serverless request operations

| Operation | Method | Purpose |
|-----------|--------|---------|
| `/runsync` | POST | Submit job, wait for result (sync) |
| `/run` | POST | Submit job, return immediately (async) |
| `/status/{id}` | GET | Check job status and results |
| `/stream/{id}` | GET | Stream incremental results |
| `/cancel/{id}` | POST | Stop a running job |
| `/health` | GET | Check endpoint health |

### Environment variable reference

| Variable | Use | Example |
|----------|-----|---------|
| `RUNPOD_API_KEY` | Authentication for SDK/CLI | `export RUNPOD_API_KEY=...` |
| `{{ RUNPOD_SECRET_name }}` | Reference encrypted secret in endpoint | `env: {"TOKEN": "{{ RUNPOD_SECRET_api_key }}"}` |
| `PORT` | Load balancer worker port | `env: {"PORT": "8000"}` |
| `HEALTH_CHECK_PATH` | Load balancer health check | `env: {"HEALTH_CHECK_PATH": "/ping"}` |

## Decision guidance

### When to use Pods vs. Serverless

| Scenario | Use Pods | Use Serverless |
|----------|----------|----------------|
| Long-running training/fine-tuning | ✓ | ✗ |
| Interactive development (JupyterLab, SSH) | ✓ | ✗ |
| Inference with variable traffic | ✗ | ✓ |
| Batch processing with queue | ✗ | ✓ |
| Predictable, continuous load | ✓ | ✗ |
| Pay-per-second, scale to zero | ✗ | ✓ |
| Custom HTTP API routes | ✗ | ✓ (load balancer) |

### When to use network volume vs. global volume

| Scenario | Network Volume | Global Volume |
|----------|----------------|---------------|
| Single datacenter workload | ✓ | ✗ |
| Multi-region access | ✗ | ✓ |
| Fast writes/POSIX compliance | ✓ | ✗ |
| Model weights shared across workers | ✓ | ✓ |
| Tied to specific data center | ✓ | ✗ |
| Region-independent storage | ✗ | ✓ |

### When to use queue-based vs. load-balanced endpoints

| Scenario | Queue-based | Load-balanced |
|----------|-------------|---------------|
| Handler function pattern | ✓ | ✗ |
| Custom HTTP routes (FastAPI, Flask) | ✗ | ✓ |
| Async job processing | ✓ | ✗ |
| Real-time HTTP API | ✗ | ✓ |
| Job queuing and status tracking | ✓ | ✗ |
| WebSocket support | ✗ | ✓ |

## Workflow

### Deploy a Serverless endpoint (queue-based)

1. **Write handler function**: Create `handler.py` with a function that accepts `job` dict, extracts `job["input"]`, processes it, and returns result.
2. **Test locally**: Run `python handler.py` or use `rp flash dev` to test with live code reloading.
3. **Create Dockerfile**: Package handler, dependencies, and base image. Use `FROM runpod/base:latest` or similar.
4. **Push to registry**: Build and push image to Docker Hub or GitHub Container Registry.
5. **Deploy endpoint**: Via console (Serverless → New Endpoint → Docker) or CLI: `runpodctl serverless create --name my-endpoint --gpu-id "NVIDIA A40" --image my-image:latest`.
6. **Configure scaling**: Set min/max workers, idle timeout, and scaling policy (queue delay or request count).
7. **Send requests**: Use `/runsync` for sync jobs or `/run` for async, with `{"input": {...}}` payload.
8. **Monitor**: Check logs, worker status, and metrics in console or via API.

### Deploy a Pod for development

1. **Choose GPU**: Decide on GPU type, region, and cloud tier (Secure or Community).
2. **Select template**: Pick a pre-configured template (PyTorch, TensorFlow, etc.) or custom image.
3. **Configure storage**: Attach network volume for persistent data or use volume disk.
4. **Deploy**: Via console (Pods → Deploy) or CLI: `runpodctl pod create --name dev-pod --gpu-id "NVIDIA RTX 4090" --image runpod/pytorch:latest`.
5. **Connect**: SSH, JupyterLab, or VS Code via console or CLI.
6. **Develop**: Install packages, run code, save work to network volume.
7. **Stop/terminate**: Stop to pause billing (storage still charged), or terminate to delete.

### Build and deploy with the SDK (Flash)

1. **Install SDK**: `pip install runpod` (requires Python 3.10+).
2. **Authenticate**: `rp login` to store credentials.
3. **Create app**: Define `runpod.App("name")` with decorated functions (`@app.queue`, `@app.task`, `@app.api`).
4. **Add local entrypoint**: Mark main function with `@runpod.local_entrypoint`.
5. **Test locally**: `rp flash dev main.py` to run functions on remote GPUs from your machine.
6. **Deploy**: `rp flash deploy` to create production endpoints.
7. **Call functions**: Use `.remote()`, `.spawn()`, or `.stream()` to invoke remote functions.

### Manage secrets and environment variables

1. **Create secret**: Via console (Settings → Secrets) or API: `POST /v2/account/secrets` with `name` and `value`.
2. **Reference in endpoint**: Set env var to `{{ RUNPOD_SECRET_name }}` in endpoint config.
3. **Access in code**: Read from `os.environ["VAR_NAME"]` in handler or Pod.
4. **Rotate**: Update secret value via API or console; new workers pick up new value.

## Common gotchas

- **Cold starts**: Serverless workers take time to initialize (load models, install packages). Minimize by using cached models, enabling FlashBoot, or keeping min workers > 0.
- **Payload size limits**: `/run` max 10 MB, `/runsync` max 20 MB. For larger results, store in S3 or network volume and return links.
- **Handler initialization**: Load models and heavy resources **outside** the handler function, not inside. Inside handler runs on every request.
- **GPU availability**: Requested GPU may be out of capacity. Use API to check availability before creating Pods, or subscribe to "deploy when available."
- **Network volume datacenter**: Network volumes are tied to one datacenter. Pod must be in same datacenter to attach. Global volumes have no datacenter constraint.
- **Secrets are write-only**: Secret values are never returned by API. Store securely and rotate periodically.
- **Worker refresh**: Long-running workers accumulate state. Use `"refresh_worker": True` in handler return to clear state after job.
- **Timeout confusion**: `executionTimeout` is max runtime once worker picks up job. `ttl` is total lifespan from submission to deletion. Both have 7-day max.
- **Rate limits scale with workers**: Base limit + (running_workers × per-worker_limit). Hitting 429? Implement backoff and retry logic.
- **Docker Compose not supported**: Runpod runs Docker for you. Cannot use Docker Compose or docker-in-docker on Pods.
- **UDP not supported**: Only TCP and HTTP connections. No UDP sockets.
- **Async handler cleanup**: Catch `asyncio.CancelledError` to release resources when job is cancelled, then re-raise.

## Verification checklist

Before submitting work:

- [ ] Handler function tested locally with `python handler.py` or `rp flash dev`.
- [ ] Dockerfile builds without errors and includes all dependencies.
- [ ] Environment variables (including secrets) are set correctly in endpoint config.
- [ ] API key has appropriate permissions (check in console under Settings → API Keys).
- [ ] Network volume (if used) is in same datacenter as Pod/endpoint.
- [ ] Request payload is under size limits (10 MB for `/run`, 20 MB for `/runsync`).
- [ ] Endpoint scaling settings (min/max workers, idle timeout) match workload.
- [ ] Logs are checked for errors: console Logs tab or `rp logs POD_ID --follow`.
- [ ] Cold start time is acceptable (test with `/runsync` first request).
- [ ] Results are retrieved within retention window (30 min async, 1 min sync).

## Resources

- **Comprehensive navigation**: https://docs.runpod.io/llms.txt
- **API v2 reference**: https://docs.runpod.io/api-reference-v2/overview
- **Serverless quickstart**: https://docs.runpod.io/serverless/quickstart
- **SDK overview and examples**: https://docs.runpod.io/sdk/overview

---

> For additional documentation and navigation, see: https://docs.runpod.io/llms.txt