Confluence Indexing API

Captain imports Confluence Cloud pages into a collection with one API call: a whole site, selected spaces, one space, or one page with the pages under it. Each page is converted to Markdown and becomes a document, with its comments and labels. Files attached to pages become documents of their own.

This is an import. It runs when called, and calling it again brings the collection up to date. Read What an import does not do before relying on it as a live mirror.

How it works

  1. An Atlassian account creates an API token.
  2. Each indexing request carries the site, that account's email and the token.
  3. Captain checks the token against Confluence before the job starts, so a wrong token usually returns an error right away. If Atlassian is slow or failing at that moment, the job is accepted unchecked and a wrong token fails the job later, with the reason in its error_message. On a site that lets anonymous visitors read some spaces, Confluence answers a wrong token as a visitor. Captain asks Confluence who the caller is and refuses an anonymous answer. A scoped token without read:user:confluence cannot be asked that question, so on such a site add that scope.
  4. The job reads every page the token can see in the requested scope and indexes it.

The email and token are encrypted before the job starts and are never logged in plain text. The encrypted token stays in that job's execution record for up to 90 days, where only the indexing pipeline can decrypt it. See Security notes.

Atlassian recommends OAuth for apps that connect to Confluence, and asks apps not to collect API tokens. OAuth support is coming. Until then, use a token from a dedicated account that can see only what should be indexed, and revoke it when the import is done.

One-time setup

Step 1: Choose the account

The job sees exactly what the token's account can see. A dedicated account (for example captain-import@yourco.com) with access to only the spaces to index keeps the scope explicit.

Step 2: Create an API token

  1. Sign in as that account at id.atlassian.com/manage-profile/security/api-tokens.

  2. Click Create API token, or Create API token with scopes for a token limited to reading Confluence.

  3. For a scoped token, select Confluence and add these scopes:

    ScopeNeeded forWithout it
    read:space:confluenceListing spacesThe request is refused, or the job fails.
    read:page:confluenceReading pagesThe request is refused, or the job fails.
    read:hierarchical-content:confluenceListing the pages under a pageA page import with its descendants fails.
    read:comment:confluenceCommentsPages are imported without comments (comments_unavailable).
    read:attachment:confluenceAttachmentsPages are imported without attachments (attachments_unavailable).
    read:label:confluenceLabels (together with read:page:confluence)Pages are imported without labels (labels_unavailable).
    read:user:confluenceAuthor and mention names, and confirming who the token belongs toNames show as account IDs (author_names_unavailable).

    The first three are required. The other four are optional: the import still succeeds, and the job's source_report.left_out says what was missing.

  4. Copy the token. Atlassian shows it once.

Both classic and scoped tokens work. A scoped token is the safer choice because it can only read.

Indexing

All three endpoints take the same three fields (confluence_site, confluence_email, confluence_api_token) and differ only in scope.

confluence_site is the site host, such as acme.atlassian.net. A full URL copied from the browser works too. Only *.atlassian.net sites are accepted.

A whole site

curl -X POST "https://api.filesearch.dev/v2/collections/{collection_name}/index/confluence" \
  -H "Authorization: Bearer $CAPTAIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "confluence_site": "acme.atlassian.net",
    "confluence_email": "captain-import@yourco.com",
    "confluence_api_token": "<Atlassian API token>",
    "processing_type": "basic"
  }'

Every space the token can see is indexed. To index only some spaces, add "space_keys": ["ENG", "OPS"].

One space

curl -X POST "https://api.filesearch.dev/v2/collections/{collection_name}/index/confluence/space" \
  -H "Authorization: Bearer $CAPTAIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "confluence_site": "acme.atlassian.net",
    "confluence_email": "captain-import@yourco.com",
    "confluence_api_token": "<Atlassian API token>",
    "space_key": "ENG",
    "processing_type": "basic"
  }'

space_key is the segment after /spaces/ in a space URL (acme.atlassian.net/wiki/spaces/ENG).

One page and the pages under it

curl -X POST "https://api.filesearch.dev/v2/collections/{collection_name}/index/confluence/page" \
  -H "Authorization: Bearer $CAPTAIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "confluence_site": "acme.atlassian.net",
    "confluence_email": "captain-import@yourco.com",
    "confluence_api_token": "<Atlassian API token>",
    "page_id": "123456789",
    "processing_type": "basic"
  }'

page_id is the number in a page URL (acme.atlassian.net/wiki/spaces/ENG/pages/123456789/Runbook). The pages under it are indexed too. Send "include_descendants": false to index only that page.

Every response is { "job_id": "...", "status": "pending" }, plus custom_metadata when the request sent it. Poll GET /v2/jobs/{job_id} for progress, then query the collection with POST /v3/collections/{name}/query.

Options

FieldDefaultWhat it does
include_attachmentstrueIndexes files attached to pages as their own documents.
include_imagesfalseAlso imports pictures, audio and video attached to pages. Requires include_attachments.
include_commentstrueAdds footer and inline comments to each page under a Comments heading.
forcefalseRe-indexes every page and attachment, including unchanged ones.
processing_typerequiredHow attachments such as PDFs are parsed: basic, advanced or auto. Pages are always indexed as text.
custom_metadatanoneMetadata attached to every chunk the job indexes.
mask_piifalseMasks personal data before it is embedded and stored. See PII Masking.

Running it again

A second call with the same scope updates the collection:

  • A page that has not changed is skipped. It costs nothing.
  • A page that has changed is replaced in place. It keeps its document ID, and its old text is no longer returned by queries. It is billed as text again.
  • A new page is added.
  • An attachment removed from a page is removed from the collection by the next import in which all of these hold: include_attachments is on, the token can list the page's attachments, and the page has 1,000 attachments or fewer. Nothing is removed between imports. This covers attachments removed from a page that still exists: when a page itself is deleted in Confluence, neither it nor its attachments are removed.

What counts as a change. A page is compared as the document it becomes. Any of these re-indexes the page:

  • its title, body or labels changed;
  • a comment was added, edited or deleted, when include_comments is on;
  • an attachment was added, removed or renamed;
  • the display name of a person it mentions, or who wrote it or commented on it, changed;
  • it was moved: its parent page, its space or its URL changed;
  • include_comments is set differently from the last import, and the page has comments.

On pages with busy comment threads, "include_comments": false avoids re-indexing, and re-billing, the page each time someone comments.

The list of a page's attachments is part of the page's text whether or not the attachments themselves are imported. With include_attachments off, their names appear as a list at the end of the page. So turning include_attachments on or off does not by itself change a page. One exception: when the token cannot list a page's attachments, the list is left out of the page's text.

A collection takes one Confluence import at a time. A call made before an earlier import into the same collection has finished returns 409 confluence_import_running with that job's job_id. A call that arrives while another one is still being submitted returns 409 confluence_import_submitting with no job ID: retry in a few seconds.

There is one behaviour, so the skip_existing and overwrite_existing fields used by other indexing endpoints are not accepted here. Sending either returns 422. To re-index everything regardless of changes, send "force": true.

What an import does not do

  • No schedule. Nothing runs until the endpoint is called. To keep a collection current, call it on a schedule from your own system.
  • No deletes yet. A page deleted or archived in Confluence stays in the collection until it is deleted there. So does a page that is restricted after it was imported: it stays searchable until it is deleted from the collection. The job result links to the list of every page it saw, so after a complete run the difference can be found and removed. See Finding deleted pages.
  • Separate from the console importer. Pages imported earlier with the Confluence importer in the console are stored under a different identity. Importing the same pages with the API adds a second copy. It does not replace the first. The one-import-at-a-time rule also covers API imports only: it cannot see an import running in the console.
  • Permissions are flattened. Confluence page and space restrictions are not carried over. Everyone who can query the collection can read everything the token could see. Use a token that sees only what the collection's readers are allowed to read.
  • Cloud only. Confluence Data Center and Server are not supported.

What gets indexed

  • Pages: each page is converted to Markdown and stored as text: the title, the body (headings, lists, tables, code blocks, links, panels, expand sections, status labels and mentions), the labels, and the comments when include_comments is true.

  • Attachments: PDFs, Office files, text files and other supported file types, each as its own document linked to its page.

  • Pictures, audio and video attached to a page are imported only with include_images: true. Each attached picture then becomes its own document, holding the text read from the image and a description of it, linked to its page. Audio and video are billed at their per-minute rates. The 250 MB size limit applies. Left out, they are reported as images_not_requested (pictures) or media_not_requested (audio and video).

  • Only files attached to a page are imported. A picture that a page shows from an external address is not downloaded. It stays in the page text as a link.

  • Metadata on every chunk. Filter on these keys in queries. A key with no value is left out. Every value is cut at 1,024 characters.

    KeyOn pagesOn attachmentsValue
    confluence_spaceyesyesThe space key.
    confluence_page_idyesyesThe page's ID. On an attachment, its page's.
    confluence_urlyesyesThe page's web address. On an attachment, its page's.
    confluence_updated_atyesyesWhen it was last changed in Confluence.
    confluence_authoryesnoThe page's author. An account ID when names are unavailable.
    confluence_labelsyesnoThe page's labels, up to 50.
    confluence_parent_idyesnoThe ID of the page above it.
    confluence_attachment_idnoyesThe attachment's ID.
    source_roleparentchildFilter on it to query pages only, or attachments only.
    source_parent_identitynoyesThe identity of the attachment's page, confluence://{site}/{page_id}. Filter on it to find a page's attachments.

    source_role and source_parent_identity are set by Captain. They can be read and filtered on. They cannot be set when indexing, and a metadata update cannot change or remove them. Sending one back with exactly its stored value on a metadata update is accepted and changes nothing.

This metadata is stored as the document's custom_metadata. The v3 documents list returns custom_metadata only when the request has include_custom_metadata=true. Without it the field is null.

Each document's uri is its identity in Confluence, not a storage location: confluence://{site}/{page_id} for a page and confluence://{site}/{page_id}/attachments/{attachment_id} for an attachment. The page's web address is in confluence_url.

Not indexed:

  • Blog posts, whiteboards and databases. A blog post's ID sent as page_id fails with a "not found" message.
  • Live docs may not be indexed.
  • Archived pages, drafts and pages in the trash. Importing one of them by page_id fails, and the job's error_message reads: "Confluence import failed: page 12345 is a draft, archived or in the trash, so it was not imported. Publish it and run the import again."
  • Pages the token cannot see. They are absent, and the job still succeeds.

Include, excerpt and Jira macros appear as a labelled reference to the page or issue they point to. The content they pull in is not copied into the page.

Limits

LimitValueWhen it is reached
Pages per job10,000 (500 during a trial)The job indexes up to the limit and reports page_cap.
Attachments per job5,000 new or changed. Attachments the collection already holds unchanged do not count. With force: true every attachment is fetched, so every one counts.The rest are left out and reported as attachment_cap. Run the job again: each run imports the next batch, so a site with more attachments is covered over several runs.
Attachments per page200 per import, counting unchanged onesA page with more imports its first 200. The rest are reported as attachment_cap.
Size of one attachment250 MBLeft out and reported as too_large.
Comments per page500The rest are not imported. Reported as comments_truncated.
Spaces in space_keys100The request is refused with 422.
Time to list the pagesabout 275 secondsThe job fails: "Confluence import failed: listing the pages did not finish in time (N pages listed). Index fewer spaces per job."
Time to download one attachmentabout 540 secondsThe attachment is left out and reported as time_budget. Run the job again to pick it up.
Attachments on a page, for tracking removals1,000On a page with more, an attachment removed in Confluence is not removed from the collection. Reported as attachment_removals_not_tracked.

Every limit is per job, not per collection or per organization. A job that reaches one indexes up to it and reports what it left out. For a larger site, index it one space at a time.

Job results

GET /v2/jobs/{job_id} returns a source_report object for these jobs:

{
  "source_report": {
    "source": "confluence",
    "pages_seen": 1200,
    "pages_new": 14,
    "pages_changed": 31,
    "pages_skipped_unchanged": 1153,
    "pages_failed": 2,
    "attachments_seen": 340,
    "attachments_new": 3,
    "attachments_changed": 1,
    "attachments_skipped_unchanged": 336,
    "attachments_failed": 0,
    "left_out": [
      { "kind": "attachment", "count": 12, "reason": "images_not_requested" }
    ],
    "identities_url": "https://..."
  }
}

left_out lists what was not indexed, or was indexed without part of its content, grouped by kind (such as page or attachment) and reason. New reasons can be added over time, so treat one that is not listed here as information, not an error.

ReasonMeaning
page_capThe job reached its page limit.
space_not_foundA space named in the request does not exist, or the token cannot see it. The job still completes, with zero pages for that space. It is not a failed job.
restrictedThe token cannot read the page, or the page was deleted or restricted while the job ran. The job goes on. If none of the listed pages can be read any more, the job fails instead and says so.
unsupported_typeThe content type, or the attachment's file type, cannot be indexed.
comments_unavailableThe page was indexed without its comments. The token is missing the read:comment:confluence scope.
labels_unavailableThe page was indexed without its labels. The token is missing read:page:confluence or read:label:confluence.
attachments_unavailableThe page was indexed without its attachments. The token is missing the read:attachment:confluence scope.
author_names_unavailableThe page was indexed with account IDs in place of author and mention names. The token is missing the read:user:confluence scope.
attachments_not_requestedAn attachment, with include_attachments off.
images_not_requestedA picture attachment, with include_images off.
media_not_requestedAn audio or video attachment, with include_images off.
attachment_capThe job reached its attachment limit.
too_largeThe attachment is over the size limit.
emptyThe attachment has no content.
download_failedThe attachment could not be downloaded from Confluence.
time_budgetThe attachment's download ran out of time. Run the job again to pick it up.
staged_copy_changedAnother import replaced the stored copy of this attachment while this job was working on it. Run the job again. A page in the same situation is counted in pages_failed, not listed here.
comments_truncatedThe page has more than 500 comments, and the rest were not imported.
attachment_removals_not_trackedThe page has more than 1,000 attachments, so one removed in Confluence is not removed from the collection.
unknownNo reason was recorded.

A job that fails before it reads any page, for example because the token was refused, has no source_report. The reason is in the job's error_message.

Finding deleted pages

identities_url is a link, valid for 15 minutes, to a text file with one line for everything the job saw: confluence://acme.atlassian.net/123456789 for a page, and confluence://acme.atlassian.net/123456789/attachments/att42 for an attachment.

The file means "exists in Confluence as far as this job could see", not "was indexed". A page is listed if the job saw it. An attachment is listed if Confluence listed it on a page the job read, whether or not it was imported: an attachment left out as too_large, images_not_requested or attachment_cap is still in the file.

A document missing from the file is therefore not always a deleted page. Use the file to remove documents only when all of these hold:

  1. The job finished with status completed. A job that failed may leave no file, or an incomplete one.

  2. pages_failed is 0. A page that failed to fetch is listed, but its attachments are not.

  3. left_out has none of these four reasons. After any of them, pages or attachments that still exist in Confluence are missing from the file:

    ReasonWhat is missing from the file
    restrictedThe page and all its attachments.
    page_capEvery page past the limit, and their attachments.
    space_not_foundEverything in that space.
    attachments_unavailableThe page's attachments. The page itself is listed, and the failed counts stay 0.
  4. The comparison covers only documents from the same scope. A job for one space says nothing about pages in other spaces, and a job for one page tree says nothing about the rest of its space. Filter the collection's documents by confluence_space before comparing. For a job scoped to one page and the pages under it there is no stored key that names that tree, so the file cannot be used to find deleted pages in it. A page MOVED to another space is absent from its old space's file and still exists. Check its ID against the other spaces before deleting it.

  5. The token is the same one, with the same access, as earlier runs. A page the token can no longer see is absent from the file but still exists.

A document in that scope whose identity is not in the file is then either gone from Confluence or moved out of the scope. Delete it with DELETE /v2/collections/{name}/documents/{document_id} once it is confirmed gone.

Billing

  • Pages are stored as Markdown and bill as text: 0.5 credit per 3,000 characters of that Markdown, rounded up for each page. A page under 3,000 characters costs 0.5 credit.
  • Attached pictures cost 0.5 credit each. Pictures, audio and video are indexed, and billed, only when include_images is true.
  • Other attachments bill at the rate for their file type and processing_type, the same as any other indexed file.
  • Skipped pages and attachments cost nothing.
  • A changed page is billed again as text. A page counts as changed when its title, body, labels, comments or list of attachments changed, or when include_comments is set differently from the last import. See What counts as a change. "include_comments": false avoids re-billing pages whenever someone comments.
  • A page is indexed again, and billed again as text, in the import that first links one of its media attachments (a picture, an audio file or a video). That is how the attachment is linked to its page. It happens the first time the attachment is indexed: when include_images is turned on for pages imported without it, and also when a media attachment that an earlier run left out, at a limit or after a failed download, is picked up by a later run. A PDF, spreadsheet or text attachment picked up later does not re-index its page.
  • A media attachment that fails to index on every run can re-bill its page. If a picture, audio file or video is downloaded but can never be indexed (a damaged file, for example), its page is indexed again, and billed again as text, each time include_images is switched on or off. The same happens whenever its download result changes from one run to the next: a run where the download fails after one where it worked, or the other way round. Runs with the same result in a row cost nothing extra. Remove or replace the file in Confluence to stop it.
  • Audio and video attachments bill at their per-minute rates.
  • Backup Protection + Storage: $0.44 per 1,000 credits stored per month. Each page and attachment stores the credits it cost to index.
  • Queries are free. Reranking is billed at $0.05 per 1M tokens.

A worked example. A space has three pages of 27,263, 2,100 and 800 characters, and one of them has a picture attached:

ItemHow it is countedCredits
27,263-character page10 blocks of 3,000 characters, rounded up5.0
2,100-character page1 block0.5
800-character page1 block0.5
Attached picture, with include_images onflat0.5
First import6.5

Those 6.5 credits are also what the documents store, so storage for them is 6.5 × $0.44 / 1,000 per month. A second import with nothing changed costs 0 credits.

Troubleshooting

SymptomCause and fix
400 "Confluence did not accept this email and API token"The token is wrong, expired or revoked, or it belongs to a different email. A scoped token missing read:space:confluence gets the same answer.
400 "The API token can list Confluence spaces ... but cannot read pages"A scoped token is missing read:page:confluence. Create a token with every scope listed in Step 2.
400 "Confluence did not recognise this email and API token ... and answered as an anonymous visitor"The site allows anonymous access and the token is wrong, so Confluence treated the request as a visitor. Check the email and the token.
400 "No Confluence site was found"No Atlassian site has that name. Check confluence_site.
400 "The API token cannot see any Confluence spaces"The account has no access to any space. Add it to the spaces to index.
422 naming skip_existing or overwrite_existingRemove the field. A second call already skips unchanged pages. Use force to re-index everything.
422 on confluence_siteOnly *.atlassian.net sites are accepted.
409 confluence_import_runningAnother Confluence import into the same collection has not finished. Wait for the job in job_id to finish, or cancel it.
409 confluence_import_submittingAnother request to import into the same collection is being submitted right now. No job ID is given, because that request may still be refused. Retry in a few seconds.
429 "Maximum concurrent indexing jobs reached"The organization is at its limit of jobs running at once, across all sources. Wait for a job to finish.
422 naming a field such as max_pagesThe field is not accepted. The limits are fixed per job and cannot be set in the request.
422 on confluence_email or confluence_api_tokenNeither may contain spaces. The token must be plain ASCII, up to 2,048 characters. The email may be up to 320 characters.
400 naming a metadata key such as source_roleCaptain sets source_role and source_parent_identity itself. They can be read and filtered on. They cannot be set on an index request, and a metadata update cannot change or remove them. Sending one back unchanged on a metadata update is accepted.
503 "could not be secured" or import_guard_unavailableThe credentials could not be encrypted, or Captain could not check whether another import is running. No job was started. Retry.
503 plan_unverifiedCaptain could not read your plan, which sets the page limit. No job was started. Retry shortly.
503 job_registry_unavailableThe job could not be recorded, so it was not started. Retry.
503 job_start_unknownThe request to start the job got no answer, so the job may or may not be running. Check the job_id in the response with GET /v2/jobs/{job_id}. If it is not found after a few minutes, it did not start and the import can be submitted again. Send a new Idempotency-Key when you do: a request with the same key replays the same job ID and starts nothing.
503 job_not_startedThe job could not be started, and nothing is running. Retry.
Job fails with an error_message that starts "Confluence import failed:"The message says why and what to do: the token was refused or revoked, a scoped token is missing a required scope, the space or page was not found, the page is a draft, the page (or every listed page) could no longer be read because it was deleted or restricted after the import started, or Confluence kept rate limiting the import. Fix that and start a new job. Pages already indexed stay indexed.
Job fails with "an internal error on our side"The fault is Captain's, not the token's. Start the job again. Unchanged pages are skipped.
Fewer pages than expectedRestricted pages the token cannot see are absent. Check source_report.left_out for a reached limit.
409 confluence_credentials_required from POST /v2/jobs/{job_id}/reindexA Confluence job cannot be retried from its job ID, because its credentials were encrypted for that one job. Start a new job. Unchanged pages are skipped.

Security notes

  • The email and token are encrypted before the job starts and never appear in plain text in job names, job status or logs.
  • The encrypted token is kept in the job's execution record for up to 90 days. Only the indexing pipeline can decrypt it, and the encryption is bound to that job and your organization. Revoke the token after the import if it is not needed again.
  • Captain only reads from Confluence. It never creates, edits or deletes pages.
  • The token is sent to two places only: your site ({site}.atlassian.net) and Atlassian's API gateway (api.atlassian.com), which scoped tokens require.
  • Revoke the token at any time at id.atlassian.com. Pages already indexed stay indexed. What a running job does depends on how Confluence answers once the token is revoked. When Confluence refuses the token outright, the job fails and says the token was refused. When it answers each remaining page as forbidden or not found instead, those pages cannot be told apart from pages the token was never allowed to see: they are reported as restricted and the job completes. If every listed page is unreadable, the job fails. After revoking a token mid-run, check the job's left_out list before trusting it as complete.
Was this page helpful?