---
name: unstructured
description: "Use when processing unstructured documents and data into
  structured formats for RAG, AI applications, and data pipelines. Agents should
  reach for this skill when: building document processing workflows, extracting
  structured data from PDFs/images/emails, chunking and embedding documents for
  vector databases, setting up automated data pipelines from cloud storage to
  vector stores, or integrating document processing into AI agents and
  applications."
metadata:
  mintlify-proj: unstructured
  version: "1.0"
---

# Unstructured Skill

## Product summary

Unstructured is a document processing platform that converts unstructured files (PDFs, images, emails, Word docs, spreadsheets, etc.) into structured JSON output. It provides three main interfaces: **Transform** (API/MCP for direct file processing), **Pipelines** (no-code platform for automated workflows), and **Open Source** (self-hosted library). 

Key entry points:
- **Transform API**: `https://transform.unstructured.io/api/v2/` — parse and extract from files
- **Pipelines API**: REST endpoints for workflow management at `https://api.unstructured.io`
- **Transform MCP**: Model Context Protocol server for AI tools (Claude, Cursor, VS Code, etc.)
- **Python SDK**: `unstructured` package for programmatic access
- **Primary docs**: https://docs.unstructured.io

Core operations: **Partition** (convert to structured elements), **Extract** (custom JSON schema), **Chunk** (split for embeddings), **Enrich** (add AI descriptions), **Embed** (generate vectors).

## When to use

Reach for this skill when:

- **Processing documents for RAG**: Partition PDFs, images, or emails into structured elements, chunk them, generate embeddings, and load into vector databases (Pinecone, Weaviate, Milvus, etc.)
- **Extracting structured data**: Define a JSON schema and extract specific fields from forms, invoices, contracts, or unstructured text
- **Building automated pipelines**: Set up workflows that ingest from cloud storage (S3, Azure, Google Drive, Dropbox) and write to vector stores or data warehouses
- **Enriching documents**: Add AI-generated image descriptions, table summaries, HTML conversions, or named entity recognition
- **One-off document processing**: Use Transform MCP in Claude or other AI tools to quickly partition a file without setup
- **Handling complex layouts**: Use VLM (Vision Language Model) partitioning for PDFs with tables, images, handwriting, or multiple languages

Do not use for: simple text parsing (use regex), database queries (use SQL), or real-time streaming (Unstructured is batch-oriented).

## Quick reference

### Partitioning strategies

| Strategy | Speed | Cost | Quality | Best for |
|----------|-------|------|---------|----------|
| **Auto** | Variable | Variable | High | Default choice; routes pages intelligently |
| **Fast** | 100x faster | Low | Medium | Text-only documents |
| **High Res** | Slower | Medium | High | Complex layouts, bounding boxes needed |
| **VLM** | Slowest | High | Highest | Images, handwriting, multilingual, tables |

### Chunking strategies

| Strategy | Use case |
|----------|----------|
| **By Title** | Preserve section boundaries; good for structured docs |
| **By Character** | Basic combining of elements up to max size |
| **By Page** | Preserve page boundaries |
| **By Similarity** | Semantic grouping using embeddings |

### File types supported

Transform and Pipelines support 65+ file types: `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.txt`, `.html`, `.eml`, `.msg`, `.jpg`, `.png`, `.tiff`, and many more. Check `transform/supported-file-types` for the complete list.

### API authentication

```bash
export UNSTRUCTURED_API_KEY="your-api-key"
export UNSTRUCTURED_API_URL="https://api.unstructured.io"  # For Pipelines
```

Get your API key from https://platform.unstructured.io → API Keys. Each key is workspace-specific.

### Transform API endpoints

| Operation | Method | Endpoint |
|-----------|--------|----------|
| Parse document | POST | `/api/v2/parse` |
| Extract with schema | POST | `/api/v2/extract` |
| Get job status | GET | `/api/v2/jobs/{job_id}` |
| List jobs | GET | `/api/v2/jobs` |

### Pipelines API endpoints

| Operation | Method | Endpoint |
|-----------|--------|----------|
| List workflows | GET | `/workflows` |
| Create workflow | POST | `/workflows` |
| Run workflow | POST | `/workflows/{id}/run` |
| List jobs | GET | `/jobs` |
| Get job details | GET | `/jobs/{id}/details` |

## Decision guidance

### When to use Transform vs Pipelines

| Scenario | Use Transform | Use Pipelines |
|----------|---------------|---------------|
| One-off file processing | ✓ | |
| Interactive AI tool (Claude, Cursor) | ✓ | |
| Automated scheduled workflows | | ✓ |
| Processing from cloud storage | | ✓ |
| Custom schema extraction | ✓ | ✓ |
| Batch processing with monitoring | | ✓ |
| No infrastructure setup needed | ✓ | |

### When to use which partitioning strategy

| Document type | Recommended |
|---------------|-------------|
| PDFs with text only | Fast |
| PDFs with tables/images | Auto or High Res |
| Scanned documents, handwriting | VLM |
| Mixed document types | Auto |
| Need bounding box coordinates | High Res or VLM |

### When to use which chunking strategy

| Goal | Strategy |
|------|----------|
| Preserve document structure | By Title |
| Simple fixed-size chunks | By Character |
| Keep pages intact | By Page |
| Semantic relevance for RAG | By Similarity |

## Workflow

### Typical document-to-vector-database pipeline

1. **Authenticate**: Set `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL` environment variables.

2. **Choose your interface**:
   - For one-off processing: Use Transform API or MCP in Claude/Cursor
   - For automated workflows: Use Pipelines platform

3. **For Transform (API/MCP)**:
   - Upload or reference a file
   - Call `/api/v2/parse` to partition (returns JSON elements)
   - Optionally call `/api/v2/extract` with a custom schema
   - Retrieve results; check `status` field for completion

4. **For Pipelines (automated)**:
   - Create a source connector (S3, Google Drive, Dropbox, etc.)
   - Create a destination connector (Pinecone, Weaviate, MongoDB, etc.)
   - Create a workflow: select source → partitioner → chunker → embedder → destination
   - Set schedule (optional) or run manually
   - Monitor job progress via `/jobs/{id}/details`

5. **Post-processing**:
   - Parse JSON output; extract `elements` array
   - Each element has `type`, `text`, `metadata` (coordinates, page, parent_id, etc.)
   - For tables: decode `text_as_html` field to render HTML
   - For images: decode `image_base64` field to recover images

6. **Verify**:
   - Check `status` field is `completed`
   - Review `warnings` array for issues
   - Validate element count and metadata presence
   - Test chunking/embedding output before loading to vector DB

### Example: Parse a PDF via Transform API

```bash
curl -X POST https://transform.unstructured.io/api/v2/parse \
  -H "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  -F "input=@document.pdf"
```

Response includes `id`, `status`, `markdown`, `elements`, `metadata`, `warnings`.

### Example: Create a Pipelines workflow (Python)

```python
from unstructured_client import UnstructuredClient
from unstructured_client.models import shared

client = UnstructuredClient(api_key="your-key", server_url="https://api.unstructured.io")

workflow = client.workflows.create_workflow(
    name="my-workflow",
    source_id="source-123",
    destination_id="dest-456",
    # Partitioner, chunker, embedder configured via UI or API
)

job = client.workflows.run_workflow(workflow_id=workflow.id)
```

## Common gotchas

- **API key scope**: Each API key is tied to one workspace. Verify you're using the correct key for your workspace.
- **Wait behavior**: Parse/Extract requests wait up to 15 minutes by default. For immediate async response, use `Prefer: wait=0` header or `wait_seconds=0` parameter.
- **File size limits**: Transform web app/API: 100 MB per file. MCP clients: 50 MB per file.
- **Partitioning strategy mismatch**: Using Fast on a PDF with complex tables will produce poor results. Use Auto or VLM instead.
- **Chunking before enrichment**: Place enrichment nodes (image descriptions, table summaries) *before* chunker nodes, or enrichments may be incomplete.
- **Embedding model dimensions**: Ensure chunk size matches embedding model dimensions. Default Azure OpenAI text-embedding-3-large uses 3072 dimensions; set max_characters accordingly.
- **Metadata field size limits**: Fields like `image_base64` and `orig_elements` can exceed database column limits (e.g., Milvus VARCHAR max 65,535 chars). Plan for compression or truncation.
- **Reprocess All behavior**: In Pipelines, checking "Reprocess All" reprocesses *all* files on every run. Uncheck to process only new/updated files.
- **Workflow layout validation**: Workflows must follow valid DAG patterns (e.g., Source → Partitioner → Destination). Invalid layouts cannot be activated.
- **Connector credentials**: Credentials are encrypted at rest. Test connectors with "Save and Test" before using in workflows.
- **Rate limits**: No public rate limit values specified. Treat as unknown; monitor quota_exceeded errors in responses.

## Verification checklist

Before submitting work with Unstructured output:

- [ ] API key is set and valid (test with a simple request)
- [ ] File type is in the supported list for your interface (Transform vs Pipelines vs Open Source)
- [ ] Partitioning strategy matches document type (Auto for mixed, VLM for complex)
- [ ] Response `status` is `completed` (not `pending` or `failed`)
- [ ] `warnings` array is empty or contains only non-critical messages
- [ ] Element count is reasonable (not 0 or unexpectedly high)
- [ ] Metadata fields present: `page_number`, `coordinates` (if High Res/VLM), `parent_id` (if nested)
- [ ] For tables: `text_as_html` field is populated and valid HTML
- [ ] For images: `image_base64` field is present and decodable
- [ ] Chunking output: chunks are within max_characters limit, overlap is applied correctly
- [ ] Embeddings: vector dimensions match your vector DB schema
- [ ] Destination connector tested and credentials verified
- [ ] Workflow DAG is valid (no orphaned nodes, correct node order)
- [ ] Job completed without errors (check `/jobs/{id}/details` for failed files)

## Resources

- **Full documentation**: https://docs.unstructured.io/llms.txt (comprehensive page-by-page navigation for agents)
- **Transform quickstart**: https://docs.unstructured.io/quickstart (Claude integration in 5 minutes)
- **Pipelines quickstart**: https://docs.unstructured.io/pipelines/quickstart (end-to-end workflow setup)
- **Partitioning strategies**: https://docs.unstructured.io/concepts/partitioning (choose the right strategy)
- **API reference**: https://docs.unstructured.io/api-reference/api/overview (all endpoints and error codes)

---

> For additional documentation and navigation, see: https://docs.unstructured.io/llms.txt