Set up Unstructured with an AI agent

name: unstructured
description: "Use when processing unstructured documents and data into
  structured formats for RAG, AI applications, and data pipelines. Agents should
  reach for this skill when: building document processing workflows, extracting
  structured data from PDFs/images/emails, chunking and embedding documents for
  vector databases, setting up automated data pipelines from cloud storage to
  vector stores, or integrating document processing into AI agents and
  applications."
metadata:
  mintlify-proj: unstructured
  version: "1.0"

Unstructured Skill

Product summary

Unstructured is a document processing platform that converts unstructured files (PDFs, images, emails, Word docs, spreadsheets, etc.) into structured JSON output. It provides three main interfaces: Transform (API/MCP for direct file processing), Pipelines (no-code platform for automated workflows), and Open Source (self-hosted library).

Key entry points:

• Transform API: https://transform.unstructured.io/api/v2/ — parse and extract from files
• Pipelines API: REST endpoints for workflow management at https://api.unstructured.io
• Transform MCP: Model Context Protocol server for AI tools (Claude, Cursor, VS Code, etc.)
• Python SDK: unstructured package for programmatic access
• Primary docs: https://docs.unstructured.io

Core operations: Partition (convert to structured elements), Extract (custom JSON schema), Chunk (split for embeddings), Enrich (add AI descriptions), Embed (generate vectors).

When to use

Reach for this skill when:

• Processing documents for RAG: Partition PDFs, images, or emails into structured elements, chunk them, generate embeddings, and load into vector databases (Pinecone, Weaviate, Milvus, etc.)
• Extracting structured data: Define a JSON schema and extract specific fields from forms, invoices, contracts, or unstructured text
• Building automated pipelines: Set up workflows that ingest from cloud storage (S3, Azure, Google Drive, Dropbox) and write to vector stores or data warehouses
• Enriching documents: Add AI-generated image descriptions, table summaries, HTML conversions, or named entity recognition
• One-off document processing: Use Transform MCP in Claude or other AI tools to quickly partition a file without setup
• Handling complex layouts: Use VLM (Vision Language Model) partitioning for PDFs with tables, images, handwriting, or multiple languages

Do not use for: simple text parsing (use regex), database queries (use SQL), or real-time streaming (Unstructured is batch-oriented).

Quick reference

Partitioning strategies

Strategy	Speed	Cost	Quality	Best for
Auto	Variable	Variable	High	Default choice; routes pages intelligently
Fast	100x faster	Low	Medium	Text-only documents
High Res	Slower	Medium	High	Complex layouts, bounding boxes needed
VLM	Slowest	High	Highest	Images, handwriting, multilingual, tables

Chunking strategies

Strategy	Use case
By Title	Preserve section boundaries; good for structured docs
By Character	Basic combining of elements up to max size
By Page	Preserve page boundaries
By Similarity	Semantic grouping using embeddings

File types supported

Transform and Pipelines support 65+ file types: .pdf, .docx, .pptx, .xlsx, .csv, .txt, .html, .eml, .msg, .jpg, .png, .tiff, and many more. Check transform/supported-file-types for the complete list.

API authentication

export UNSTRUCTURED_API_KEY="your-api-key"
export UNSTRUCTURED_API_URL="https://api.unstructured.io"  # For Pipelines

Get your API key from https://platform.unstructured.io → API Keys. Each key is workspace-specific.

Transform API endpoints

Operation	Method	Endpoint
Parse document	POST	/api/v2/parse
Extract with schema	POST	/api/v2/extract
Get job status	GET	/api/v2/jobs/{job_id}
List jobs	GET	/api/v2/jobs

Pipelines API endpoints

Operation	Method	Endpoint
List workflows	GET	/workflows
Create workflow	POST	/workflows
Run workflow	POST	/workflows/{id}/run
List jobs	GET	/jobs
Get job details	GET	/jobs/{id}/details

Decision guidance

When to use Transform vs Pipelines

Scenario	Use Transform	Use Pipelines
One-off file processing	✓
Interactive AI tool (Claude, Cursor)	✓
Automated scheduled workflows		✓
Processing from cloud storage		✓
Custom schema extraction	✓	✓
Batch processing with monitoring		✓
No infrastructure setup needed	✓

When to use which partitioning strategy

Document type	Recommended
PDFs with text only	Fast
PDFs with tables/images	Auto or High Res
Scanned documents, handwriting	VLM
Mixed document types	Auto
Need bounding box coordinates	High Res or VLM

When to use which chunking strategy

Goal	Strategy
Preserve document structure	By Title
Simple fixed-size chunks	By Character
Keep pages intact	By Page
Semantic relevance for RAG	By Similarity

Workflow

Typical document-to-vector-database pipeline

1. Authenticate: Set UNSTRUCTURED_API_KEY and UNSTRUCTURED_API_URL environment variables.

2. Choose your interface:

   • For one-off processing: Use Transform API or MCP in Claude/Cursor
   • For automated workflows: Use Pipelines platform

3. For Transform (API/MCP):

   • Upload or reference a file
   • Call /api/v2/parse to partition (returns JSON elements)
   • Optionally call /api/v2/extract with a custom schema
   • Retrieve results; check status field for completion

4. For Pipelines (automated):

   • Create a source connector (S3, Google Drive, Dropbox, etc.)
   • Create a destination connector (Pinecone, Weaviate, MongoDB, etc.)
   • Create a workflow: select source → partitioner → chunker → embedder → destination
   • Set schedule (optional) or run manually
   • Monitor job progress via /jobs/{id}/details

5. Post-processing:

   • Parse JSON output; extract elements array
   • Each element has type, text, metadata (coordinates, page, parent_id, etc.)
   • For tables: decode text_as_html field to render HTML
   • For images: decode image_base64 field to recover images

6. Verify:

   • Check status field is completed
   • Review warnings array for issues
   • Validate element count and metadata presence
   • Test chunking/embedding output before loading to vector DB

Example: Parse a PDF via Transform API

curl -X POST https://transform.unstructured.io/api/v2/parse \
  -H "unstructured-api-key: $UNSTRUCTURED_API_KEY" \
  -F "input=@document.pdf"

Response includes id, status, markdown, elements, metadata, warnings.

Example: Create a Pipelines workflow (Python)

from unstructured_client import UnstructuredClient
from unstructured_client.models import shared

client = UnstructuredClient(api_key="your-key", server_url="https://api.unstructured.io")

workflow = client.workflows.create_workflow(
    name="my-workflow",
    source_id="source-123",
    destination_id="dest-456",
    # Partitioner, chunker, embedder configured via UI or API
)

job = client.workflows.run_workflow(workflow_id=workflow.id)

Common gotchas

• API key scope: Each API key is tied to one workspace. Verify you're using the correct key for your workspace.
• Wait behavior: Parse/Extract requests wait up to 15 minutes by default. For immediate async response, use Prefer: wait=0 header or wait_seconds=0 parameter.
• File size limits: Transform web app/API: 100 MB per file. MCP clients: 50 MB per file.
• Partitioning strategy mismatch: Using Fast on a PDF with complex tables will produce poor results. Use Auto or VLM instead.
• Chunking before enrichment: Place enrichment nodes (image descriptions, table summaries) before chunker nodes, or enrichments may be incomplete.
• Embedding model dimensions: Ensure chunk size matches embedding model dimensions. Default Azure OpenAI text-embedding-3-large uses 3072 dimensions; set max_characters accordingly.
• Metadata field size limits: Fields like image_base64 and orig_elements can exceed database column limits (e.g., Milvus VARCHAR max 65,535 chars). Plan for compression or truncation.
• Reprocess All behavior: In Pipelines, checking "Reprocess All" reprocesses all files on every run. Uncheck to process only new/updated files.
• Workflow layout validation: Workflows must follow valid DAG patterns (e.g., Source → Partitioner → Destination). Invalid layouts cannot be activated.
• Connector credentials: Credentials are encrypted at rest. Test connectors with "Save and Test" before using in workflows.
• Rate limits: No public rate limit values specified. Treat as unknown; monitor quota_exceeded errors in responses.

Verification checklist

Before submitting work with Unstructured output:

• API key is set and valid (test with a simple request)
• File type is in the supported list for your interface (Transform vs Pipelines vs Open Source)
• Partitioning strategy matches document type (Auto for mixed, VLM for complex)
• Response status is completed (not pending or failed)
• warnings array is empty or contains only non-critical messages
• Element count is reasonable (not 0 or unexpectedly high)
• Metadata fields present: page_number, coordinates (if High Res/VLM), parent_id (if nested)
• For tables: text_as_html field is populated and valid HTML
• For images: image_base64 field is present and decodable
• Chunking output: chunks are within max_characters limit, overlap is applied correctly
• Embeddings: vector dimensions match your vector DB schema
• Destination connector tested and credentials verified
• Workflow DAG is valid (no orphaned nodes, correct node order)
• Job completed without errors (check /jobs/{id}/details for failed files)

Resources

• Full documentation: https://docs.unstructured.io/llms.txt (comprehensive page-by-page navigation for agents)
• Transform quickstart: https://docs.unstructured.io/quickstart (Claude integration in 5 minutes)
• Pipelines quickstart: https://docs.unstructured.io/pipelines/quickstart (end-to-end workflow setup)
• Partitioning strategies: https://docs.unstructured.io/concepts/partitioning (choose the right strategy)
• API reference: https://docs.unstructured.io/api-reference/api/overview (all endpoints and error codes)

---

For additional documentation and navigation, see: https://docs.unstructured.io/llms.txt