> For a complete page index of the Captain API documentation, fetch https://docs.captain.dev/llms.txt?excludeSpec=true # Multimodal Search > Search across text, images, video, and audio in a single query. Captain uses native multimodal embeddings with cross-modal reranking for state-of-the-art retrieval quality. ## Agent Quick Reference - Multimodal Search * **Endpoint**: `POST /v2/collections/{name}/query` * **Reranking**: Multimodal collections rerank by default. Pass `rerank=false` to turn it off — it is honoured, and the response carries a `warnings` entry noting that cross-modal ranking quality is reduced. Reranking adds roughly 200 ms. * **Media processing**: Video segmented into ≤120s clips, audio into ≤80s clips, images natively embedded. * **Response fields per result**: `score`, `content`, `document_id`, `filename`, `uri`, `chunk_index`, `rerank_score`, `metadata`. The modality lives at `metadata.modality` ("text"|"pdf"|"image"|"video"|"audio"). Media segment time ranges are the structured fields `metadata.startSec` / `metadata.endSec` (seconds); the `content` label is just `[Video: file.mp4]` with no timestamps. * **Indexing**: Same endpoints as text. Captain auto-detects file type. No special configuration needed. Captain supports native multimodal search across text documents, images, video, and audio files. Upload any combination of file types to a single collection, and search across all of them with one query. Captain handles format detection, media segmentation, embedding, and cross-modal ranking automatically. ## Benchmark Results On [MRAG-Bench](https://github.com/runcaptain/captain-mrag-bench) (ICLR 2025), a standardized academic benchmark for vision-centric retrieval with 16,130 images and 1,251 questions, Captain achieves **81.3% retrieval accuracy**-outperforming every end-to-end RAG system tested in the paper, including GPT-4o with CLIP retrieval (68.96%). Full results, methodology, and evaluation code are available in our [open-source evaluation repository](https://github.com/runcaptain/captain-mrag-bench). ## Supported File Types | Modality | Formats | Processing | | ------------- | ---------------------------------------------- | ------------------------------------------------- | | **Documents** | PDF, DOCX, DOC, TXT, MD, JSON, YAML, CSV, XLSX | Text extraction, chunking, and semantic embedding | | **Images** | PNG, JPEG, GIF, BMP, TIFF, WEBP | Native multimodal embedding + visual description | | **Video** | MP4, MOV, AVI, MKV, WEBM, FLV, WMV | Segmented into ≤120s clips, natively embedded | | **Audio** | MP3, WAV, AAC, FLAC, M4A, OGG, WMA | Segmented into ≤80s clips, natively embedded | ## How It Works Each image, video clip, or audio clip is stored as one record with two kinds of signal: 1. **A native embedding of the media itself.** The image pixels, the video frames, or the audio waveform are embedded directly. This is what a query matches on when the answer is in how something looks or sounds. 2. **Searchable text next to that vector.** A model writes a description of what the clip shows or contains, and speech is transcribed. Both are stored on the same record as full-text fields, so a query can match a word that was said, a caption on a slide, or a label on a part. The description never paraphrases speech; spoken words live in the transcript. A query for a song therefore finds a track by the sound of the music and by the transcribed lyrics at the same time. A search for the line "uptown funk you up" surfaces the right track even when the file is named `track03.mp3`. The per-type guides describe what each description contains: [Images](/guides/file-types/images), [Video](/guides/file-types/video), [Audio](/guides/file-types/audio). ### Why reranking matters for cross-modal results Relevance scores from text search and media search are produced by different models on different scales. A text reranker score of 0.7 and a media cosine similarity of 0.7 do not mean the same thing and cannot be sorted into a single list. Captain solves this with **reranker-informed pipeline weighting**: the text reranker's scores on media descriptions determine how much weight each modality gets in the final ranking. Without it, results from different modalities are ordered by their own pipeline scores, which are not comparable. Because of that, multimodal collections rerank by default and you do not need to set `rerank=true` yourself. You can turn it off with `rerank=false` when latency matters more than cross-modal ordering — a voice assistant on a single-modality corpus, for example. Reranking costs roughly 200 ms. ### Text-only collections If your collection contains only text documents, multimodal search adds zero overhead. Captain automatically detects whether a collection has media content and skips the multimodal pipeline entirely for text-only collections. ## Querying Multimodal Collections > **Note** > > Multimodal collections (images, video, or audio) rerank by default, so you do not need to set `rerank=true`. Passing `rerank=false` turns it off and returns a `warnings` entry noting reduced cross-modal ranking quality. Text-only collections honour the flag exactly as sent. ### Example: Multimodal Search ```python import requests import uuid BASE_URL = "https://api.filesearch.dev" API_KEY = "your_api_key" headers = { "Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json", "Idempotency-Key": str(uuid.uuid4()) } response = requests.post( f"{BASE_URL}/v2/collections/my_media_library/query", headers=headers, json={ "query": "product demo with live dashboard", "rerank": True, "top_k": 10 }, timeout=120.0 ) data = response.json() for result in data["search_results"]: modality = result["metadata"].get("modality", "text") print(f"[{modality}] {result['filename']} | score: {result['score']:.3f}") # For video/audio results, the segment time range is in metadata.startSec / # metadata.endSec (seconds). meta = result["metadata"] if meta.get("startSec") is not None: print(f" segment {meta['startSec']:.0f}s-{meta['endSec']:.0f}s") print(f" {result['content'][:100]}") ``` **Response:** ```json { "success": true, "search_results": [ { "score": 0.95, "content": "[Video: product_demo.mp4]\nIn the video, a presenter walks through the analytics dashboard showing real-time revenue metrics and user engagement charts...", "document_id": "doc_video_123", "filename": "product_demo.mp4", "uri": "s3://my-media-bucket/videos/product_demo.mp4", "chunk_index": 1, "metadata": { "modality": "video", "source": "multimodal", "startSec": 120.0, "endSec": 240.0 } }, { "score": 0.88, "content": "[Image: dashboard_screenshot.png]\n### Visual Description\nA screenshot of a web application dashboard with a dark theme. The main area displays a line chart of monthly revenue trending upward...", "document_id": "doc_img_456", "filename": "dashboard_screenshot.png", "uri": "s3://my-media-bucket/images/dashboard_screenshot.png", "chunk_index": 0, "metadata": { "modality": "image", "source": "multimodal" } }, { "score": 0.82, "content": "The dashboard provides real-time analytics including revenue metrics, user engagement, and conversion rates...", "document_id": "doc_pdf_789", "filename": "product_docs.pdf", "uri": "s3://my-company-docs/product_docs.pdf", "chunk_index": 15, "metadata": { "modality": "text", "source": "hybrid", "pageStart": 8, "pageEnd": 8 } } ], "total_results": 3, "top_k": 10, "query": "product demo with live dashboard", "execution_time_ms": 1240 } ``` ### Request Fields | Field | Type | Default | Description | | ----------------- | ------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `query` | string | *required* | The natural language search query | | `rerank` | boolean | `false` (`true` for multimodal) | Enable reranking. Multimodal collections default to `true`; pass `false` to opt out. Text-only collections use the value as sent. Adds roughly 200 ms | | `top_k` | integer | `10` | Number of results to return | | `metadata_filter` | object | `null` | Filter expression (`$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$and`, `$or`) | ### Response Fields (per search result) | Field | Type | Description | | ----------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `score` | number | Final relevance score | | `content` | string | Text content, or media label + VLM description (e.g., `[Video: file.mp4]\nDescription...`) | | `document_id` | string | Unique identifier of the source file | | `filename` | string | Name of the source file | | `uri` | string \| null | Original source URI of the file | | `chunk_index` | integer | Index of this chunk (0 for images, segment number for video/audio) | | `rerank_score` | number \| null | Reranker score when `rerank=true`, otherwise null | | `metadata.modality` | string | Content type: `text`, `pdf`, `image`, `video`, or `audio` | | `metadata.source` | string | How this result was found: `hybrid` (text), `vector` (text), `bm25` (text), or `multimodal` (media) | | `metadata.pageStart` | integer \| null | Starting page number (text/PDF only) | | `metadata.pageEnd` | integer \| null | Ending page number (text/PDF only) | | `metadata.startSec` | number \| null | Start of the segment in seconds (video/audio only) | | `metadata.endSec` | number \| null | End of the segment in seconds (video/audio only) | | `metadata.transcriptLanguage` | string \| null | Language the transcript was recognized in, as an AWS Transcribe code (e.g. `es-ES`) — auto-detected or set via `transcription_language` at indexing time. `null` for non-media results and for video/audio indexed before language support | | `metadata.sheetName` | string \| null | Worksheet name the chunk came from (spreadsheets only) | | `metadata.section` | string \| null | Logical table section within the sheet (e.g. a repeated-header or stacked-table block); may carry a label like `section 1 (2024)` | | `metadata.rowStart` | integer \| null | First worksheet row of the chunk, 1-based (spreadsheets only) | | `metadata.rowEnd` | integer \| null | Last worksheet row of the chunk, 1-based (spreadsheets only) | | `metadata.colStart` | integer \| null | First column of the chunk, 1-based (spreadsheets only) | | `metadata.colEnd` | integer \| null | Last column of the chunk, 1-based (spreadsheets only) | | `metadata.columns` | string\[] \| null | Header labels for the chunk's columns, in order (spreadsheets only) | | `metadata.rowRole` | string \| null | `aggregate` when the chunk's rows are totals/subtotals/averages, otherwise null (spreadsheets only) | ### Spreadsheet row/column provenance For spreadsheet sources (XLSX, XLS, CSV, TSV), each chunk is a structure-aware block of rows rendered as a markdown table, and carries first-class **row/column provenance** - the tabular analogue of `pageStart`/`pageEnd` for PDFs. This lets you ground an answer in concrete coordinates, e.g. *"Sheet 'YTD Sales', rows 2–13, columns Year–Ytd Total Sales"*. ```json { "score": 0.81, "content": "# Sheet: YTD Sales - section 1 (2024)\n| Year | Month | Weekly Net Sales | Ytd Total Sales |\n| --- | --- | --- | --- |\n| 2024 | Jan | 8025 | 339987 |", "document_id": "doc_xyz789", "filename": "sales.xlsx", "chunk_index": 0, "metadata": { "sheetName": "YTD Sales", "section": "section 1 (2024)", "rowStart": 2, "rowEnd": 13, "colStart": 1, "colEnd": 4, "columns": ["Year", "Month", "Weekly Net Sales", "Ytd Total Sales"], "rowRole": null } } ``` These fields are `null` for non-spreadsheet sources, exactly as `pageStart`/`pageEnd` are `null` for spreadsheets. The chunk's `content` repeats the header row, so the table is self-describing even when read in isolation. ### Media segment time range For video and audio, each result is one **segment** of the source file. Its time range is returned as the structured fields **`metadata.startSec`** and **`metadata.endSec`** (seconds), the time analogue of `pageStart`/`pageEnd` for documents. Read these directly; do not parse the `content` string. ```json { "score": 0.88, "content": "[Video: product_demo.mp4]\nThe presenter demonstrates the checkout flow...", "metadata": { "modality": "video", "startSec": 120.0, "endSec": 158.4 } } ``` The `content` field's first line is a citation **label** (`[Video: ]`) followed by the VLM description and (for video) the spoken transcript. The label no longer carries the time range. `startSec`/`endSec` are `null` for non-media (text/PDF) results, just as `pageStart`/`pageEnd` are `null` for media. | Modality | `content` label | Example | | -------- | --------------------- | ---------------------------- | | Video | `[Video: ]` | `[Video: product_demo.mp4]` | | Audio | `[Audio: ]` | `[Audio: earnings_call.mp3]` | | Image | `[Image: ]` | `[Image: architecture.png]` | If you pass `rerank=false` on a multimodal collection, reranking is skipped and the response carries a `warnings` entry so the quality trade-off is visible: ```json { "success": true, "warnings": [ "Reranking is disabled on a collection containing multimodal content. Cross-modal ranking quality is reduced: results from different modalities are ordered by their pipeline scores rather than by a single reranker." ], "search_results": [ ... ] } ``` Results are still returned from every modality. Only the ordering across modalities is affected. Text-only collections honor `rerank=false` as before. ## Indexing Media Files Media files are indexed through the same endpoints as text documents. No special configuration is needed-Captain automatically detects the file type and routes it through the appropriate processing pipeline. ### Transcription language (video and audio) Speech in video and audio files is transcribed with **automatic language detection** by default — Spanish interviews come back as Spanish text, Portuguese as Portuguese, and so on, with no configuration. Detection runs on the whole file's audio, so it is reliable even for short clips. When you already know the spoken language (or a file mixes languages and you want to force the dominant one), pass the optional **`transcription_language`** field on any indexing request, as an [AWS Transcribe language code](https://docs.aws.amazon.com/transcribe/latest/dg/supported-languages.html) such as `es-US`, `es-ES`, `pt-BR`, or `en-US`: ```python response = requests.post( f"{BASE_URL}/v2/collections/consumer_interviews/index/url", headers=headers, json={ "url": "https://example.com/entrevista-consumidor.mp4", "processing_type": "advanced", "transcription_language": "es-US" } ) ``` * Omit the field to auto-detect (recommended for most workloads). * Codes are case-insensitive (`es-us` is accepted and canonicalized to `es-US`); an unsupported value is rejected with a `422` listing example codes. * The field applies to video and audio files only; other file types in the same request ignore it. * Each result's `metadata.transcriptLanguage` reports the language the transcript was actually recognized in. * Files indexed before language support (or while a former English-only default applied) can be re-indexed with `overwrite_existing: true` to regenerate their transcripts. ### Index from cloud storage ```python import requests BASE_URL = "https://api.filesearch.dev" API_KEY = "your_api_key" headers = { "Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json" } # Index an S3 bucket containing mixed content (PDFs, images, videos, audio) response = requests.post( f"{BASE_URL}/v2/collections/my_media_library/index/s3", headers=headers, json={ "bucket_name": "my-media-bucket", "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE", "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", "bucket_region": "us-east-1", "processing_type": "advanced" } ) print(response.json()) ``` ### Index from URL ```python # Index media files from direct URLs response = requests.post( f"{BASE_URL}/v2/collections/my_media_library/index/url", headers=headers, json={ "urls": [ "https://example.com/product-demo.mp4", "https://example.com/screenshot.png", "https://example.com/podcast-episode.mp3" ], "processing_type": "basic" } ) ``` ### Upload files directly ```python # Upload files via multipart form with open("meeting_recording.mp4", "rb") as video, open("notes.pdf", "rb") as pdf: response = requests.post( f"{BASE_URL}/v2/collections/my_media_library/index/file", headers={ "Authorization": f"Bearer {API_KEY}", }, files=[ ("files", ("meeting_recording.mp4", video, "video/mp4")), ("files", ("notes.pdf", pdf, "application/pdf")), ], data={"processing_type": "basic"} ) ``` ## Using Retrieved Media Chunks The query endpoint returns retrieved chunks across all content types. Your application can pass those chunks to an LLM, cite them directly, or store chunk IDs for later metadata and relation work. ```python response = requests.post( f"{BASE_URL}/v2/collections/my_media_library/query", headers=headers, json={ "query": "What was discussed about the product roadmap?", "rerank": True, "top_k": 5 }, timeout=120.0 ) data = response.json() for result in data["search_results"]: print(result["document_id"], result["content"][:160]) ``` Media results use labels such as `[Video: meeting_recording.mp4]` followed by the VLM-generated description, with the segment time range in `metadata.startSec`/`metadata.endSec`. ## Limitations * **Video segments**: Maximum 120 seconds per segment. Longer videos are automatically split. * **Mixed-language media**: Language detection picks one dominant language per file. For video or audio that switches languages mid-file, force the dominant one with `transcription_language`. * **Audio segments**: Maximum 80 seconds per segment. Longer audio files are automatically split. * **Format conversion**: Some formats are converted at ingestion (FLAC→MP3, WebP→PNG, AVI→MP4). The original file is preserved. * **Reranking**: Multimodal collections rerank by default. Passing `rerank=false` turns it off and returns a `warnings` entry about reduced cross-modal ranking quality. Text-only collections are unaffected. * **Latency**: Multimodal queries take \~1-2 seconds (text embedding + media search + reranking). Reranking is roughly 200 ms of that and can be turned off with `rerank=false`. Text-only queries are unchanged.