> For a complete page index of the Captain API documentation, fetch https://docs.captain.dev/llms.txt?excludeSpec=true # Set Up Sync > Connect a cloud storage source to a Captain collection and keep its files up to date automatically. Add one source or several, and Captain keeps the collection current as files are added, changed, and removed. > **Info** > > **Set up once, stay current.** A sync connects a cloud storage bucket to a Captain collection. Captain indexes what is already there, then keeps the collection current as files are added, changed, and removed. ## Agent Quick Reference: Sync * A sync binds one bucket/prefix to one collection. Create it and Captain backfills the bucket, then keeps the collection current as objects are added, changed, and removed. * Providers: AWS S3, Google Cloud Storage, Azure Blob Storage, Cloudflare R2, Supabase Storage, Backblaze B2. Create endpoints are per provider (`sync/s3`, `sync/gcs`, `sync/azure`, `sync/r2`, `sync/supabase`, `sync/backblaze`); management endpoints are shared. * Auth by provider: S3 uses assume-role (`type=assume_role` with `role_arn` + `external_id`, recommended) or access-key (`type=access_key`). R2/Supabase/Backblaze use S3-compatible access keys (R2 passes `account_id` + `jurisdiction`; Supabase and Backblaze pass `endpoint_url`). GCS uses `service_account_json` (a service account key with read access). Azure Blob uses `container` + `account_name` + `account_key` (a storage account key; the container is echoed back as `bucket`, and the key can be rotated later with `PATCH /v2/syncs/{sync_id}` `account_key`). Secrets are stored securely and never returned. The assume-role external ID is issued by Captain (email [humans@captain.dev](mailto:humans@captain.dev)). * Two mechanisms keep the collection current: scheduled reconciliation (`sync_interval_minutes`, minimum 5; values below 5 are raised to 5; `null`/omitted means manual) and optional real-time events (via `webhooks`). Reconciliation is the backstop for any dropped events. * Real-time enrollment differs by provider: S3 passes `{"sns_topic_arn": ...}` and subscribes Captain's returned `queue_arn` to that topic; every other provider enrolls with an empty body, receives a private `ingest_url`, and points the store's change notifications at it (Pub/Sub push for GCS, Event Grid subscription for Azure Blob with a Deploy to Azure template or one `az eventgrid event-subscription create`, Database Webhook on `storage.objects` for Supabase, event notification rule for B2, queue + forwarder Worker for R2). * Change detection compares each object's ETag (a content hash), so an in-place overwrite of the same byte size is still caught. * Endpoints (base `https://api.filesearch.dev`): * `POST /v2/collections/{collection_name}/sync/s3` create + backfill * `GET /v2/syncs` list * `GET /v2/syncs/{sync_id}` get * `PATCH /v2/syncs/{sync_id}` update prefix/globs/deletion_policy/cadence/status * `POST /v2/syncs/{sync_id}/webhooks` enroll events * `POST /v2/syncs/{sync_id}/reconcile` on-demand reconcile * `DELETE /v2/syncs/{sync_id}` stop + remove the sync * `deletion_policy`: `mirror` (default, remove from collection), `archive` (keep the document but mark it archived: excluded from search by default, surfaced with `include: { archived: true }`), `ignore` (leave indexed copy). * Pause without deleting: `PATCH ... {"status": "paused"}`; resume with `{"status": "active"}`. * Google Drive sync is **Studio only**: sign in with Google in the dashboard, pick a folder, enable "keep in sync". There is no API endpoint to create or list it (it does not appear in `GET /v2/syncs`); the indexing jobs it launches do appear in `GET /v2/jobs`. Changes are detected by polling Drive on the cadence (minimum 5 minutes, default 15); no real-time events. For API-driven Drive indexing use the one-time [Google Drive Indexing API](/guides/google-drive-headless) with a service account. ## How Sync works **One setup, then hands-off.** Point a sync at a bucket. Captain indexes the existing files, then tracks the source so your search results always reflect what is in storage. **Many sources, one collection.** Set up a sync for each bucket you want included, and Captain keeps all of them current in the same collection. Each sync tracks its own source independently, so a change or deletion in one source only affects the files that came from it. **Two mechanisms keep a collection current:** * **Scheduled reconciliation** runs on the cadence you choose. Captain lists the source, compares it against what it has already indexed, and applies the difference: new files get indexed, changed files get re-indexed, and removed files follow your deletion policy. * **Real-time events** are an optional add-on. Forward change notifications from your source and updates land within seconds. Delivery is best-effort, so reconciliation always stays on as the backstop. Events are enrolled per sync with one call, `POST /v2/syncs/{sync_id}/webhooks`; each provider section below shows its exact wiring, and the [shared settings](#sync-settings-on-every-provider) (cadence, deletion policy, pause, on-demand reconcile) work identically everywhere. ## Choose your provider Six storage providers with one behavior, plus Google Drive from the Studio. Pick yours: #### [AWS S3](#aws-s3) #### [Google Cloud Storage](#google-cloud-storage) #### [Azure Blob Storage](#azure-blob-storage) #### [Cloudflare R2](#cloudflare-r2) #### [Supabase Storage](#supabase-storage) #### [Backblaze B2](#backblaze-b2) #### [Google Drive](#google-drive) Studio only --- ## AWS S3 | | | | ----------------------- | --------------------------------------------------------------------------------- | | **Authentication** | IAM assume-role (recommended) or access key | | **Real-time transport** | Your SNS topic delivers to a Captain queue. AWS-native, no URL to host | | **One-click events** | [CloudFormation Launch Stack](#one-click-setup-launch-stack) | | **Credentials guide** | [Connect Cloud Storage: AWS S3](/guides/connect-cloud-storage#aws-s3-access-keys) | ### Authentication An AWS S3 sync can authenticate two ways. **Role assumption (recommended).** Create an IAM role in your own account that grants read access to the bucket, and Captain reads through that role using a Captain-issued external ID. No long-lived keys leave your account. The setup is the same as S3 indexing, so follow the [S3 Cross-Account IAM](/guides/s3-cross-account-iam) guide, then pass an auth block of type `assume_role` with your `role_arn` and `external_id`. **Access key.** Pass an auth block of type `access_key` with an `access_key_id` and `secret_access_key`. Captain stores the secret securely and never returns it. Use this when a role is not an option. > **Info** > > The external ID for role assumption is issued by Captain for your organization. > Email [humans@captain.dev](mailto:humans@captain.dev) or your account manager to get it before you create a > sync. ### Create a sync Creating a sync starts the first full index of the collection and returns the sync record right away. The indexing itself runs in the background. ```python import requests BASE_URL = "https://api.filesearch.dev" API_KEY = "cap_your_api_key" resp = requests.post( f"{BASE_URL}/v2/collections/company-knowledge/sync/s3", headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}, json={ "bucket": "my-company-docs", "prefix": "knowledge-base/", "region": "us-east-1", "auth": { "type": "assume_role", "role_arn": "arn:aws:iam::123456789012:role/captain-s3-read", "external_id": "captain-abc123", }, "processing_type": "advanced", "deletion_policy": "mirror", "sync_interval_minutes": 15, }, ) print(resp.json()) ``` ### Real-time events For AWS S3, events run entirely inside AWS: your bucket publishes to an SNS topic in your account, and that topic delivers straight to a queue on Captain's side. There's no HTTP endpoint to host and no signature handshake to configure. #### One-click setup (Launch Stack) A CloudFormation quick-create link stands up the whole wiring below in one pass, in your own AWS account: the SNS topic, the bucket's event notification, and the enrollment with Captain. [![Launch Stack](https://raw.githubusercontent.com/runcaptain/captain-sync-templates/main/assets/launch-stack.png)](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/quickcreate?stackName=captain-s3-sync\&templateURL=https://captain-templates.s3.amazonaws.com/templates/2026-08-13/captain-s3-sync.yaml) The template is open source: [github.com/runcaptain/captain-sync-templates](https://github.com/runcaptain/captain-sync-templates). Launch it in the same region as your bucket (edit the `region=` segment of the link to match) and have your sync id and Captain API key ready; the stack prompts for them as parameters. #### Manual setup 1. Create an SNS topic in the bucket's region. 2. On the bucket, add an event notification for `s3:ObjectCreated:*` and `s3:ObjectRemoved:*` (scoped to your sync prefix if you set one) with the topic as its destination. 3. Enroll the sync with your topic ARN. Captain binds the topic to this sync and returns the queue to subscribe it to: ```python resp = requests.post( f"{BASE_URL}/v2/syncs/{sync_id}/webhooks", headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}, json={"sns_topic_arn": "arn:aws:sns:us-east-1:123456789012:my-bucket-events"}, ) print(resp.json()["queue_arn"]) print(resp.json()["instructions"]) ``` 4. Subscribe the returned `queue_arn` to your topic (protocol `sqs`). The subscription confirms automatically; there's nothing to click through. Leave raw message delivery off (the default): Captain matches your events to this sync by the topic ARN carried in the SNS envelope, so raw delivery strips the routing information and your events are ignored. Only events published through the topic you enrolled reach your sync. Events sit alongside scheduled reconciliation rather than replacing it. Delivery is at-least-once and can drop or reorder messages, so keep a [cadence](#cadence) set even with events enabled. --- ## Google Cloud Storage | | | | ----------------------- | -------------------------------------------------------------------------------------------- | | **Authentication** | Service-account JSON with read access (`roles/storage.objectViewer`) | | **Real-time transport** | Pub/Sub push subscription to your `ingest_url` | | **One-click events** | Three `gcloud` commands | | **Credentials guide** | [Connect Cloud Storage: GCS](/guides/connect-cloud-storage#google-cloud-storage-credentials) | ### Authentication A GCS sync authenticates with a **service account key**. Create a service account with read access to the bucket (`roles/storage.objectViewer`), download its JSON key, and pass the whole key as `service_account_json` when you create the sync. Captain stores the key securely and never returns it. ### Create a sync ```python resp = requests.post( f"{BASE_URL}/v2/collections/company-knowledge/sync/gcs", headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}, json={ "bucket": "my-bucket", "prefix": "docs/", "service_account_json": open("service-account.json").read(), "processing_type": "advanced", "deletion_policy": "mirror", "sync_interval_minutes": 15, }, ) print(resp.json()) ``` GCS buckets are global, so there is no `region` or `endpoint_url` to pass. ### Real-time events GCS publishes bucket changes to Pub/Sub, and Pub/Sub pushes them straight to Captain. Enroll first: Captain returns an `ingest_url`, an HTTPS endpoint hosted by Captain that listens for this sync's notifications. ```python resp = requests.post( f"{BASE_URL}/v2/syncs/{sync_id}/webhooks", headers={"Authorization": f"Bearer {API_KEY}"}, ) print(resp.json()["ingest_url"]) print(resp.json()["instructions"]) ``` Then wire the bucket to it: ```bash # 1. A topic for the bucket's change notifications gcloud pubsub topics create my-captain-events # 2. Publish object creations and deletions to it (scoped to your prefix if set) gsutil notification create -t my-captain-events -f json \ -e OBJECT_FINALIZE -e OBJECT_DELETE gs://my-bucket # 3. Push the notifications to your ingest_url gcloud pubsub subscriptions create my-captain-events-sub \ --topic my-captain-events --push-endpoint="" --ack-deadline=30 ``` `OBJECT_FINALIZE` fires for new and overwritten files and triggers a re-index; `OBJECT_DELETE` follows your deletion policy. Metadata-only changes are ignored. Keep the `ingest_url` private, because it authenticates the event stream for your sync. --- ## Azure Blob Storage | | | | ----------------------- | ------------------------------------------------------------------------------------------------- | | **Authentication** | Storage account name + access key, rotatable in place | | **Real-time transport** | Event Grid subscription to your `ingest_url` | | **One-click events** | [Deploy to Azure template](#one-click-setup-deploy-to-azure) | | **Credentials guide** | [Connect Cloud Storage: Azure Blob](/guides/connect-cloud-storage#azure-blob-storage-credentials) | ### Authentication An Azure Blob sync authenticates with the storage account's **name and access key** (key1 or key2 from the account's Access keys blade). Pass them as `account_name` and `account_key` when creating the sync. Captain stores the key securely and never returns it. When the key is rotated on the Azure side, hand Captain the new one with `PATCH /v2/syncs/{sync_id}` and an `account_key` field: the new key is validated against the container before it replaces the stored one. ### Create a sync The container name is passed as `container` and echoed back as `bucket` in sync responses. ```python resp = requests.post( f"{BASE_URL}/v2/collections/company-knowledge/sync/azure", headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}, json={ "container": "policy-documents", "account_name": "contosodocs", "account_key": "", "prefix": "docs/", "processing_type": "advanced", "deletion_policy": "mirror", "sync_interval_minutes": 60, }, ) print(resp.json()) ``` Azure storage accounts are addressed by name, so there is no `region` or `endpoint_url` to pass. `sync_interval_minutes` defaults to 60 for Azure so reconciliation backstops any missed event delivery. ### Real-time events Azure publishes blob changes through Event Grid, which delivers them straight to Captain over HTTPS. Enroll first: Captain returns an `ingest_url`, an HTTPS endpoint hosted by Captain that listens for this sync's notifications. ```python resp = requests.post( f"{BASE_URL}/v2/syncs/{sync_id}/webhooks", headers={"Authorization": f"Bearer {API_KEY}"}, ) print(resp.json()["ingest_url"]) print(resp.json()["instructions"]) ``` #### One-click setup (Deploy to Azure) A Deploy to Azure template stands up the whole wiring in one pass, in your own Azure subscription: the Event Grid system topic on the storage account, the event subscription pointed at your `ingest_url`, and a self-verifying enrollment check. The deployment only goes green if Event Grid proves it can deliver to Captain and Captain confirms the webhook is enrolled on your sync. [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fcaptaindeploytemplates.blob.core.windows.net%2Ftemplates%2F2026-09-07%2Fcaptain-blob-sync.json) The template is open source: [github.com/runcaptain/captain-sync-templates](https://github.com/runcaptain/captain-sync-templates). Deploy it into the resource group that holds the storage account, in the same region, with your sync id, Captain API key, and `ingest_url` ready; the template prompts for them as parameters. #### Manual setup One az CLI command creates the same subscription: ```bash az eventgrid event-subscription create \ --name captain-sync \ --source-resource-id $(az storage account show -n contosodocs -g --query id -o tsv) \ --endpoint "" --endpoint-type webhook \ --included-event-types Microsoft.Storage.BlobCreated Microsoft.Storage.BlobDeleted \ --subject-begins-with "/blobServices/default/containers/policy-documents/blobs/docs/" \ --event-delivery-schema eventgridschema ``` Azure validates the endpoint during the create (a 30-second handshake with Captain), so a failed create means the `ingest_url` is wrong or stale. `BlobCreated` fires for new and overwritten blobs and triggers a re-index; `BlobDeleted` follows your deletion policy. Renames on hierarchical-namespace (ADLS Gen2) accounts converge on the next reconcile. Keep the `ingest_url` private, because it authenticates the event stream for your sync; if it leaks, rotate it with `{"rotate_secret": true}` on the webhooks call, then update the event subscription's endpoint to the new URL (which re-runs the handshake). --- ## Cloudflare R2 | | | | ----------------------- | ------------------------------------------------------------------------------------ | | **Authentication** | R2 API token (S3-compatible Access Key ID + Secret) + account ID | | **Real-time transport** | R2 bucket notifications to a Queue, forwarded to your `ingest_url` by a small Worker | | **One-click events** | Two `wrangler` commands + a 20-line forwarder Worker (Workers Paid plan) | | **Credentials guide** | [Connect Cloud Storage: R2](/guides/connect-cloud-storage#cloudflare-r2-credentials) | ### Authentication R2 speaks the S3 API with its own keys: create an **R2 API token** (Access Key ID + Secret Access Key). R2 derives its endpoint from your account ID and jurisdiction, so pass `account_id` rather than a full URL. For a US-jurisdiction bucket, set `jurisdiction` to `us`. In Studio Sync Manager, select **US** under **Jurisdiction**. This selects the bucket endpoint; it does not move the bucket or change its jurisdiction. ### Create a sync ```python resp = requests.post( f"{BASE_URL}/v2/collections/company-knowledge/sync/r2", headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}, json={ "bucket": "my-bucket", "prefix": "docs/", "account_id": "your-account-id", "access_key_id": "your-r2-key-id", "secret_access_key": "your-r2-secret", "jurisdiction": "default", # "default", "eu", "fedramp", or "us" "processing_type": "advanced", "sync_interval_minutes": 15, }, ) ``` ### Real-time events Enroll first to get your `ingest_url`: ```python resp = requests.post( f"{BASE_URL}/v2/syncs/{sync_id}/webhooks", headers={"Authorization": f"Bearer {API_KEY}"}, ) print(resp.json()["ingest_url"]) print(resp.json()["instructions"]) ``` Cloudflare only delivers R2 bucket notifications to a Queue inside your own account, and only a Worker in your account can read that queue; nothing outside Cloudflare can subscribe to it. So the event path needs one small piece on your side: bucket notifications land in your queue, and a forwarder Worker passes each one to your `ingest_url`, where Captain applies the change. Queues require the Workers Paid plan. ```bash npx wrangler queues create captain-sync-events npx wrangler r2 bucket notification create my-bucket \ --event-type object-create --event-type object-delete \ --prefix "docs/" --queue captain-sync-events ``` Then deploy the forwarder. `wrangler.toml`: ```toml name = "captain-r2-forwarder" main = "worker.js" compatibility_date = "2025-01-01" [[queues.consumers]] queue = "captain-sync-events" max_batch_size = 10 max_batch_timeout = 5 [vars] INGEST_URL = "" ``` `worker.js`: ```js export default { async queue(batch, env) { for (const msg of batch.messages) { try { const res = await fetch(env.INGEST_URL, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(msg.body), }); if (res.status >= 500) msg.retry(); else msg.ack(); } catch (e) { msg.retry(); } } }, }; ``` Run `npx wrangler deploy` and changes in the bucket start flowing to your collection. --- ## Supabase Storage | | | | ----------------------- | ------------------------------------------------------------------------------------- | | **Authentication** | S3 connection keys (Access Key ID + Secret) + project S3 endpoint | | **Real-time transport** | Database Webhook on `storage.objects` posting to your `ingest_url` | | **One-click events** | Dashboard steps (Integrations, then Database Webhooks) | | **Credentials guide** | [Connect Cloud Storage: Supabase](/guides/connect-cloud-storage#supabase-credentials) | ### Authentication Supabase Storage speaks the S3 API through a per-project endpoint. Get the **S3 endpoint**, an **Access Key ID**, and a **Secret Access Key** under **Storage → S3 Connection**, then pass the endpoint as `endpoint_url`. ### Create a sync ```python resp = requests.post( f"{BASE_URL}/v2/collections/company-knowledge/sync/supabase", headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}, json={ "bucket": "my-bucket", "prefix": "docs/", "endpoint_url": "https://PROJECT_REF.storage.supabase.co/storage/v1/s3", "access_key_id": "your-key-id", "secret_access_key": "your-secret", "processing_type": "advanced", "sync_interval_minutes": 15, }, ) ``` ### Real-time events Enroll first to get your `ingest_url`: ```python resp = requests.post( f"{BASE_URL}/v2/syncs/{sync_id}/webhooks", headers={"Authorization": f"Bearer {API_KEY}"}, ) print(resp.json()["ingest_url"]) print(resp.json()["instructions"]) ``` Supabase file changes appear as rows in the `storage.objects` table, so events come from a Database Webhook rather than the Storage UI. In the dashboard, open Integrations, install Database Webhooks if the project doesn't have it yet, and create a webhook with: * Table: **`objects`** in schema **`storage`**. Not `buckets`, which only fires when a bucket itself is created or deleted. * Events: INSERT, UPDATE, and DELETE. * Type: HTTP Request, method POST, URL: your `ingest_url`. Each file change now posts directly to the Captain-hosted `ingest_url`, and Captain applies it to the collection. The webhook fires for every bucket in the project; Captain ignores events from buckets other than the one your sync covers. --- ## Backblaze B2 | | | | ----------------------- | --------------------------------------------------------------------------------------------- | | **Authentication** | Application key (keyID + applicationKey), bucket-scoped | | **Real-time transport** | B2 event notification rule posting to your `ingest_url` | | **One-click events** | Console steps (Event Notifications on the bucket) | | **Credentials guide** | [Connect Cloud Storage: Backblaze B2](/guides/connect-cloud-storage#backblaze-b2-credentials) | ### Authentication B2 speaks the S3 API with application keys. Create an **Application Key** (keyID + applicationKey) under **Account → Application Keys**, using a bucket-scoped key rather than the master key. The **endpoint** and **region** appear on the bucket's details page. ### Create a sync ```python resp = requests.post( f"{BASE_URL}/v2/collections/company-knowledge/sync/backblaze", headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}, json={ "bucket": "my-bucket", "prefix": "docs/", "endpoint_url": "https://s3.us-west-004.backblazeb2.com", "region": "us-west-004", "access_key_id": "your-key-id", "secret_access_key": "your-app-key", "processing_type": "advanced", "sync_interval_minutes": 15, }, ) ``` ### Real-time events Enroll first to get your `ingest_url`: ```python resp = requests.post( f"{BASE_URL}/v2/syncs/{sync_id}/webhooks", headers={"Authorization": f"Bearer {API_KEY}"}, ) print(resp.json()["ingest_url"]) print(resp.json()["instructions"]) ``` Enable Event Notifications for your Backblaze account first (in the console); the API rejects rules until the feature is on. Then create an event notification rule on the bucket, in the console or via `b2_set_bucket_notification_rules`: * Event types: `b2:ObjectCreated:*` and `b2:ObjectDeleted:*`. * Prefix: your sync prefix, if set. * Target: webhook, with your `ingest_url` as the URL. B2 sends a test event when the rule is saved; Captain accepts it, so validation passes without side effects. --- ## Google Drive > **Warning** > > **Studio only.** Google Drive sync is set up and managed in the Captain Studio > (the dashboard). It is not available headlessly: there is no API endpoint to > create, list, or update a Drive sync, and Drive syncs do not appear in > `GET /v2/syncs`. If you need Drive indexing driven from code, use the one-time > [Google Drive Indexing API](/guides/google-drive-headless), which reads Drive > through a service account and needs no sign-in. | | | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | **Authentication** | Sign in with Google in the Studio (read-only Drive access). Captain keeps a durable connection, so you sign in once per Google account | | **Change detection** | Captain polls Drive for changes on the cadence you set (minimum 5 minutes, default 15) | | **Real-time events** | Not available. Polling only | | **Where to manage it** | Studio → Sync Manager, alongside your bucket syncs | ### Sign in and import 1. In the Studio, open **Home**, choose the collection and environment you want the files in, and pick **Google Drive** as the source. 2. **Sign in with Google.** Captain asks for read-only access to Drive. The connection is durable: you will not be asked again for this Google account unless you revoke access or sign out of Google Drive in the Studio. 3. **Pick a folder or files** in the Google picker. Picking a folder indexes everything in it, including subfolders; picking individual files indexes those files and treats their folder as the sync scope. 4. The import runs as an indexing job. Native Google Docs, Slides, and Sheets are exported at fetch time (Docs and Slides as PDF, Sheets as a spreadsheet) so they parse like any other document. ### Keep it in sync When the import finishes, the Studio asks **"Keep this folder in sync?"** | Setting | What it does | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Sync cadence** | Minutes between checks. Minimum 5, default 15. Captain polls Drive for changes on this interval | | **Deletion policy** | **Mirror** removes a file from search when it is deleted or trashed in Drive. **Archive** keeps the indexed copy but marks it archived (surface it with `include: { archived: true }` on a v3 query) | Enable it and the folder is watched from then on. On every check Captain asks Drive what changed since the last one and applies only the difference: * **New files** in the folder (or any subfolder) are indexed. * **Modified files** are re-indexed in place. The updated content replaces the earlier version of the same document, so a file edited ten times is still one document in the collection. * **Deleted or trashed files** follow the deletion policy. * Files that did not change are not touched, so a check on a quiet folder costs nothing. A file that fails to fetch on one check (a transient Drive error, for example) is retried on later checks; the Sync Manager shows how many retries are pending. ### Manage the sync Open **Sync Manager** in the Studio. Google Drive syncs sit next to your bucket syncs with the same columns: source, collection, status, cadence, last sync, and files updated. From the row menu you can: * **Sync now**: run a check immediately instead of waiting for the next tick. * **Edit**: change the cadence or the deletion policy. * **Pause** and **Resume**: stop checking without losing the configuration. * **Delete**: stop the sync and forget its change tracking. Already indexed documents stay in the collection. Each row carries an activity band showing the running or latest indexing job. Expand it to see recent jobs with their file counts and status, or open one in **Activity** for per-file detail. These jobs are ordinary indexing jobs, so they are also visible through `GET /v2/jobs`. ### If Google access lapses If the Google account's access is revoked, or the sign-in expires, the sync pauses and its status reads **Reconnect required**. Nothing already indexed is affected. Click **Reconnect** on the Google Drive integration card (or on the row in Sync Manager), sign in again, and checking resumes from where it left off. --- ## Create a sync in the Studio **Sync Manager** in Captain Studio creates the same syncs through the same endpoints, for all six storage providers plus Google Drive. The create form asks for what the create call takes (provider credentials, collection, prefix, patterns, processing type, cadence, deletion policy), plus: * **Environment**. Syncs are organization-wide, not scoped to an environment; the collection they fill is. The Development / Staging / Production choice picks which environment's collections are offered as the target. * **Tags**. Key/value rows sent as `custom_metadata` and attached to every document the sync indexes, filterable at query time. * **Azure Blob**. The container, storage account name, and account key (`container`, `account_name`, `account_key`). The key is never shown back and can be rotated later from the sync's **Edit** dialog. > **Warning** > > Creating a sync needs an API key in the target environment, because the indexing jobs a > sync launches are attributed to an organization key. Without one the API answers `412`, > and the Studio says "No production API key yet" (or the environment you chose). Create a > key under **API Keys** for that environment, then create the sync. Each row's **Webhook** dialog enrolls real-time events and shows the wiring for that provider (the SNS topic for S3, the private `ingest_url` and setup commands for the rest). **Rotate secret** in that dialog is the `rotate_secret` option described under [Rotate the webhook secret](#rotate-the-webhook-secret). ## Sync settings on every provider Everything below behaves identically on all six storage providers; the sections above cover only what differs (credentials, the create call, event wiring). Google Drive syncs use the same cadence, deletion policy, pause, resume, and delete semantics, but as Sync Manager controls in the Studio rather than API calls; **Sync now** is the on-demand reconcile. ### Cadence `sync_interval_minutes` sets how often scheduled reconciliation runs. > **Warning** > > The minimum cadence is 5 minutes. The scheduler runs on a single five-minute > tick, so a sync cannot reconcile more often than that. A value below 5 is > accepted and raised to 5 rather than rejected. Any value of 5 or more is used as > given, for example 15, 60, or 1440 for a daily sync. Read the sync back to > see the value in effect. Set `sync_interval_minutes` to an explicit `null` for a manual sync with no scheduled reconciliation. Leaving the field out means the provider's default: manual on every provider except Azure Blob, which defaults to a 60-minute cadence so reconciliation backstops missed event deliveries. Real-time events and on-demand reconcile work either way. When changes need to land faster than every five minutes, use real-time events rather than a shorter cadence. The cadence is a backstop interval, not a latency guarantee. Change the cadence at any time: ```python requests.patch( f"{BASE_URL}/v2/syncs/{sync_id}", headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}, json={"sync_interval_minutes": 60}, ) ``` ### Change detection Reconciliation decides what changed by comparing each object's ETag, a content hash, against the value Captain recorded when it last indexed the object. It doesn't compare file size, so an in-place overwrite that keeps the same byte count still gets caught and re-indexed. Objects whose content hasn't changed are skipped, so each pass only touches what actually changed. ### Deletion propagation When an object is no longer in the bucket, `deletion_policy` decides what happens to its indexed copy: | Policy | Behavior | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mirror` (default) | Remove it from the collection so it no longer appears in search. | | `archive` | Keep the document's content but mark it archived. Archived content is excluded from search by default and does not count against normal results; pass `include: { archived: true }` on a v3 query to surface it (on v2, `include_archived: true`). | | `ignore` | Leave the indexed copy in place. | Removed objects are matched by their exact bucket and key, so deletion targets the right document. `archive` is a one-way transition for a removed object: once its source object is gone from the bucket, the archived copy stays archived. Re-uploading the same key later starts a new live document at that key and does not un-archive the prior copy, so a query with `include: { archived: true }` can return both: the current live version and the earlier archived one. ### Scoping and parsing `prefix` scopes a sync to part of the bucket (an empty prefix syncs everything), `include_patterns` and `exclude_patterns` are glob filters over object keys, and `processing_type` picks the parsing tier (`advanced` for full document understanding, `basic` for faster and cheaper processing, `auto` to pick per file: `advanced` for files with photos, drawings, tables or scanned pages, `basic` for the rest). All four are accepted on every provider's create call. ### Reconcile on demand Run a reconciliation right away instead of waiting for the next scheduled tick. It returns the changes it found and applied. ```python resp = requests.post( f"{BASE_URL}/v2/syncs/{sync_id}/reconcile", headers={"Authorization": f"Bearer {API_KEY}"}, ) print(resp.json()) # {"sync_id": "...", "job_id": "...", "added": 0, "modified": 1, # "removed": 0, "unchanged": 0, "deleted_documents": 0} ``` ### Rotate the webhook secret The `ingest_url` a non-S3 sync receives embeds a signing secret. If it leaks, call [`POST /v2/syncs/{sync_id}/webhooks`](/reference/sync/manage/webhooks) again with `rotate_secret: true`. Captain revokes the current URL and returns a fresh one; the old endpoint stops accepting events immediately, so point the store's change notifications at the new URL right after. In the Studio this is **Rotate secret** in the sync's Webhook dialog. ```python resp = requests.post( f"{BASE_URL}/v2/syncs/{sync_id}/webhooks", headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}, json={"rotate_secret": True}, ) print(resp.json()["ingest_url"]) ``` ### Pause, resume, and delete Pause a sync to stop syncing without losing its configuration: send a `PATCH` to `/v2/syncs/{sync_id}` with `{"status": "paused"}`, and resume with `{"status": "active"}`. Delete a sync with `DELETE /v2/syncs/{sync_id}`. This is a soft delete: syncing stops, the record is marked inactive, and its per-object sync state is cleared. The history is kept. ## How it fits together > **Note** > > Captain runs the scheduled tick for you. Once a sync has a cadence, Captain > reconciles it on that interval without you calling anything. Captain never runs > anything inside your account; the pieces that live there (an SNS topic, a Pub/Sub > subscription, an Event Grid subscription, the R2 forwarder Worker) are created > and controlled by you. A common production setup: create the sync with a cadence such as `15` and `deletion_policy: mirror`, enroll the event webhook for faster updates, and leave scheduled reconciliation on as the backstop. Reconciliation catches any events that were dropped and brings the collection back in line with the bucket. ## Endpoint reference Full request and response schemas are in the [Sync section of the API Reference](/reference/sync/s3/create). At a glance: | Endpoint | Purpose | | ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | [`POST /v2/collections/{collection_name}/sync/s3`](/reference/sync/s3/create) | Create a sync and start the first index (also `/sync/gcs`, `/sync/azure`, `/sync/r2`, `/sync/supabase`, `/sync/backblaze`) | | [`GET /v2/syncs`](/reference/sync/manage/list) | List syncs | | [`GET /v2/syncs/{sync_id}`](/reference/sync/manage/get) | Get a sync and its schedule state | | [`PATCH /v2/syncs/{sync_id}`](/reference/sync/manage/update) | Update prefix, filters, deletion policy, cadence, or status | | [`POST /v2/syncs/{sync_id}/webhooks`](/reference/sync/manage/webhooks) | Enroll real-time events | | [`POST /v2/syncs/{sync_id}/reconcile`](/reference/sync/manage/reconcile) | Reconcile on demand | | [`DELETE /v2/syncs/{sync_id}`](/reference/sync/manage/delete) | Stop and remove a sync |