Confluence Indexing API
Captain imports Confluence Cloud pages into a collection with one API call: a whole site, selected spaces, one space, or one page with the pages under it. Each page is converted to Markdown and becomes a document, with its comments and labels. Files attached to pages become documents of their own.
This is an import. It runs when called, and calling it again brings the collection up to date. Read What an import does not do before relying on it as a live mirror.
How it works
- An Atlassian account creates an API token.
- Each indexing request carries the site, that account's email and the token.
- Captain checks the token against Confluence before the job starts, so a
wrong token usually returns an error right away. If Atlassian is slow or
failing at that moment, the job is accepted unchecked and a wrong token
fails the job later, with the reason in its
error_message. On a site that lets anonymous visitors read some spaces, Confluence answers a wrong token as a visitor. Captain asks Confluence who the caller is and refuses an anonymous answer. A scoped token withoutread:user:confluencecannot be asked that question, so on such a site add that scope. - The job reads every page the token can see in the requested scope and indexes it.
The email and token are encrypted before the job starts and are never logged in plain text. The encrypted token stays in that job's execution record for up to 90 days, where only the indexing pipeline can decrypt it. See Security notes.
Atlassian recommends OAuth for apps that connect to Confluence, and asks apps not to collect API tokens. OAuth support is coming. Until then, use a token from a dedicated account that can see only what should be indexed, and revoke it when the import is done.
One-time setup
Step 1: Choose the account
The job sees exactly what the token's account can see. A dedicated account
(for example captain-import@yourco.com) with access to only the spaces to
index keeps the scope explicit.
Step 2: Create an API token
-
Sign in as that account at id.atlassian.com/manage-profile/security/api-tokens.
-
Click Create API token, or Create API token with scopes for a token limited to reading Confluence.
-
For a scoped token, select Confluence and add these scopes:
Scope Needed for Without it read:space:confluenceListing spaces The request is refused, or the job fails. read:page:confluenceReading pages The request is refused, or the job fails. read:hierarchical-content:confluenceListing the pages under a page A page import with its descendants fails. read:comment:confluenceComments Pages are imported without comments ( comments_unavailable).read:attachment:confluenceAttachments Pages are imported without attachments ( attachments_unavailable).read:label:confluenceLabels (together with read:page:confluence)Pages are imported without labels ( labels_unavailable).read:user:confluenceAuthor and mention names, and confirming who the token belongs to Names show as account IDs ( author_names_unavailable).The first three are required. The other four are optional: the import still succeeds, and the job's
source_report.left_outsays what was missing. -
Copy the token. Atlassian shows it once.
Both classic and scoped tokens work. A scoped token is the safer choice because it can only read.
Indexing
All three endpoints take the same three fields (confluence_site,
confluence_email, confluence_api_token) and differ only in scope.
confluence_site is the site host, such as acme.atlassian.net. A full URL
copied from the browser works too. Only *.atlassian.net sites are accepted.
A whole site
curl -X POST "https://api.filesearch.dev/v2/collections/{collection_name}/index/confluence" \
-H "Authorization: Bearer $CAPTAIN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"confluence_site": "acme.atlassian.net",
"confluence_email": "captain-import@yourco.com",
"confluence_api_token": "<Atlassian API token>",
"processing_type": "basic"
}'Every space the token can see is indexed. To index only some spaces, add
"space_keys": ["ENG", "OPS"].
One space
curl -X POST "https://api.filesearch.dev/v2/collections/{collection_name}/index/confluence/space" \
-H "Authorization: Bearer $CAPTAIN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"confluence_site": "acme.atlassian.net",
"confluence_email": "captain-import@yourco.com",
"confluence_api_token": "<Atlassian API token>",
"space_key": "ENG",
"processing_type": "basic"
}'space_key is the segment after /spaces/ in a space URL
(acme.atlassian.net/wiki/spaces/ENG).
One page and the pages under it
curl -X POST "https://api.filesearch.dev/v2/collections/{collection_name}/index/confluence/page" \
-H "Authorization: Bearer $CAPTAIN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"confluence_site": "acme.atlassian.net",
"confluence_email": "captain-import@yourco.com",
"confluence_api_token": "<Atlassian API token>",
"page_id": "123456789",
"processing_type": "basic"
}'page_id is the number in a page URL
(acme.atlassian.net/wiki/spaces/ENG/pages/123456789/Runbook). The pages
under it are indexed too. Send "include_descendants": false to index only
that page.
Every response is { "job_id": "...", "status": "pending" }, plus
custom_metadata when the request sent it. Poll
GET /v2/jobs/{job_id} for progress, then query the collection with
POST /v3/collections/{name}/query.
Options
| Field | Default | What it does |
|---|---|---|
include_attachments | true | Indexes files attached to pages as their own documents. |
include_images | false | Also imports pictures, audio and video attached to pages. Requires include_attachments. |
include_comments | true | Adds footer and inline comments to each page under a Comments heading. |
force | false | Re-indexes every page and attachment, including unchanged ones. |
processing_type | required | How attachments such as PDFs are parsed: basic, advanced or auto. Pages are always indexed as text. |
custom_metadata | none | Metadata attached to every chunk the job indexes. |
mask_pii | false | Masks personal data before it is embedded and stored. See PII Masking. |
Running it again
A second call with the same scope updates the collection:
- A page that has not changed is skipped. It costs nothing.
- A page that has changed is replaced in place. It keeps its document ID, and its old text is no longer returned by queries. It is billed as text again.
- A new page is added.
- An attachment removed from a page is removed from the collection by the
next import in which all of these hold:
include_attachmentsis on, the token can list the page's attachments, and the page has 1,000 attachments or fewer. Nothing is removed between imports. This covers attachments removed from a page that still exists: when a page itself is deleted in Confluence, neither it nor its attachments are removed.
What counts as a change. A page is compared as the document it becomes. Any of these re-indexes the page:
- its title, body or labels changed;
- a comment was added, edited or deleted, when
include_commentsis on; - an attachment was added, removed or renamed;
- the display name of a person it mentions, or who wrote it or commented on it, changed;
- it was moved: its parent page, its space or its URL changed;
include_commentsis set differently from the last import, and the page has comments.
On pages with busy comment threads, "include_comments": false avoids
re-indexing, and re-billing, the page each time someone comments.
The list of a page's attachments is part of the page's text whether or not
the attachments themselves are imported. With include_attachments off,
their names appear as a list at the end of the page. So turning
include_attachments on or off does not by itself change a page. One
exception: when the token cannot list a page's attachments, the list is left
out of the page's text.
A collection takes one Confluence import at a time. A call made before an
earlier import into the same collection has finished returns 409 confluence_import_running with that job's job_id. A call that arrives
while another one is still being submitted returns 409 confluence_import_submitting with no job ID: retry in a few seconds.
There is one behaviour, so the skip_existing and overwrite_existing
fields used by other indexing endpoints are not accepted here. Sending either
returns 422. To re-index everything regardless of changes, send
"force": true.
What an import does not do
- No schedule. Nothing runs until the endpoint is called. To keep a collection current, call it on a schedule from your own system.
- No deletes yet. A page deleted or archived in Confluence stays in the collection until it is deleted there. So does a page that is restricted after it was imported: it stays searchable until it is deleted from the collection. The job result links to the list of every page it saw, so after a complete run the difference can be found and removed. See Finding deleted pages.
- Separate from the console importer. Pages imported earlier with the Confluence importer in the console are stored under a different identity. Importing the same pages with the API adds a second copy. It does not replace the first. The one-import-at-a-time rule also covers API imports only: it cannot see an import running in the console.
- Permissions are flattened. Confluence page and space restrictions are not carried over. Everyone who can query the collection can read everything the token could see. Use a token that sees only what the collection's readers are allowed to read.
- Cloud only. Confluence Data Center and Server are not supported.
What gets indexed
-
Pages: each page is converted to Markdown and stored as text: the title, the body (headings, lists, tables, code blocks, links, panels, expand sections, status labels and mentions), the labels, and the comments when
include_commentsis true. -
Attachments: PDFs, Office files, text files and other supported file types, each as its own document linked to its page.
-
Pictures, audio and video attached to a page are imported only with
include_images: true. Each attached picture then becomes its own document, holding the text read from the image and a description of it, linked to its page. Audio and video are billed at their per-minute rates. The 250 MB size limit applies. Left out, they are reported asimages_not_requested(pictures) ormedia_not_requested(audio and video). -
Only files attached to a page are imported. A picture that a page shows from an external address is not downloaded. It stays in the page text as a link.
-
Metadata on every chunk. Filter on these keys in queries. A key with no value is left out. Every value is cut at 1,024 characters.
Key On pages On attachments Value confluence_spaceyes yes The space key. confluence_page_idyes yes The page's ID. On an attachment, its page's. confluence_urlyes yes The page's web address. On an attachment, its page's. confluence_updated_atyes yes When it was last changed in Confluence. confluence_authoryes no The page's author. An account ID when names are unavailable. confluence_labelsyes no The page's labels, up to 50. confluence_parent_idyes no The ID of the page above it. confluence_attachment_idno yes The attachment's ID. source_roleparentchildFilter on it to query pages only, or attachments only. source_parent_identityno yes The identity of the attachment's page, confluence://{site}/{page_id}. Filter on it to find a page's attachments.source_roleandsource_parent_identityare set by Captain. They can be read and filtered on. They cannot be set when indexing, and a metadata update cannot change or remove them. Sending one back with exactly its stored value on a metadata update is accepted and changes nothing.
This metadata is stored as the document's custom_metadata. The v3 documents
list returns custom_metadata only when the request has
include_custom_metadata=true. Without it the field is null.
Each document's uri is its identity in Confluence, not a storage location:
confluence://{site}/{page_id} for a page and
confluence://{site}/{page_id}/attachments/{attachment_id} for an
attachment. The page's web address is in confluence_url.
Not indexed:
- Blog posts, whiteboards and databases. A blog post's ID sent as
page_idfails with a "not found" message. - Live docs may not be indexed.
- Archived pages, drafts and pages in the trash. Importing one of them by
page_idfails, and the job'serror_messagereads: "Confluence import failed: page 12345 is a draft, archived or in the trash, so it was not imported. Publish it and run the import again." - Pages the token cannot see. They are absent, and the job still succeeds.
Include, excerpt and Jira macros appear as a labelled reference to the page or issue they point to. The content they pull in is not copied into the page.
Limits
| Limit | Value | When it is reached |
|---|---|---|
| Pages per job | 10,000 (500 during a trial) | The job indexes up to the limit and reports page_cap. |
| Attachments per job | 5,000 new or changed. Attachments the collection already holds unchanged do not count. With force: true every attachment is fetched, so every one counts. | The rest are left out and reported as attachment_cap. Run the job again: each run imports the next batch, so a site with more attachments is covered over several runs. |
| Attachments per page | 200 per import, counting unchanged ones | A page with more imports its first 200. The rest are reported as attachment_cap. |
| Size of one attachment | 250 MB | Left out and reported as too_large. |
| Comments per page | 500 | The rest are not imported. Reported as comments_truncated. |
Spaces in space_keys | 100 | The request is refused with 422. |
| Time to list the pages | about 275 seconds | The job fails: "Confluence import failed: listing the pages did not finish in time (N pages listed). Index fewer spaces per job." |
| Time to download one attachment | about 540 seconds | The attachment is left out and reported as time_budget. Run the job again to pick it up. |
| Attachments on a page, for tracking removals | 1,000 | On a page with more, an attachment removed in Confluence is not removed from the collection. Reported as attachment_removals_not_tracked. |
Every limit is per job, not per collection or per organization. A job that reaches one indexes up to it and reports what it left out. For a larger site, index it one space at a time.
Job results
GET /v2/jobs/{job_id} returns a source_report object for these jobs:
{
"source_report": {
"source": "confluence",
"pages_seen": 1200,
"pages_new": 14,
"pages_changed": 31,
"pages_skipped_unchanged": 1153,
"pages_failed": 2,
"attachments_seen": 340,
"attachments_new": 3,
"attachments_changed": 1,
"attachments_skipped_unchanged": 336,
"attachments_failed": 0,
"left_out": [
{ "kind": "attachment", "count": 12, "reason": "images_not_requested" }
],
"identities_url": "https://..."
}
}left_out lists what was not indexed, or was indexed without part of its
content, grouped by kind (such as page or attachment) and reason. New
reasons can be added over time, so treat one that is not listed here as
information, not an error.
| Reason | Meaning |
|---|---|
page_cap | The job reached its page limit. |
space_not_found | A space named in the request does not exist, or the token cannot see it. The job still completes, with zero pages for that space. It is not a failed job. |
restricted | The token cannot read the page, or the page was deleted or restricted while the job ran. The job goes on. If none of the listed pages can be read any more, the job fails instead and says so. |
unsupported_type | The content type, or the attachment's file type, cannot be indexed. |
comments_unavailable | The page was indexed without its comments. The token is missing the read:comment:confluence scope. |
labels_unavailable | The page was indexed without its labels. The token is missing read:page:confluence or read:label:confluence. |
attachments_unavailable | The page was indexed without its attachments. The token is missing the read:attachment:confluence scope. |
author_names_unavailable | The page was indexed with account IDs in place of author and mention names. The token is missing the read:user:confluence scope. |
attachments_not_requested | An attachment, with include_attachments off. |
images_not_requested | A picture attachment, with include_images off. |
media_not_requested | An audio or video attachment, with include_images off. |
attachment_cap | The job reached its attachment limit. |
too_large | The attachment is over the size limit. |
empty | The attachment has no content. |
download_failed | The attachment could not be downloaded from Confluence. |
time_budget | The attachment's download ran out of time. Run the job again to pick it up. |
staged_copy_changed | Another import replaced the stored copy of this attachment while this job was working on it. Run the job again. A page in the same situation is counted in pages_failed, not listed here. |
comments_truncated | The page has more than 500 comments, and the rest were not imported. |
attachment_removals_not_tracked | The page has more than 1,000 attachments, so one removed in Confluence is not removed from the collection. |
unknown | No reason was recorded. |
A job that fails before it reads any page, for example because the token was
refused, has no source_report. The reason is in the job's error_message.
Finding deleted pages
identities_url is a link, valid for 15 minutes, to a text file with one line
for everything the job saw: confluence://acme.atlassian.net/123456789 for a
page, and confluence://acme.atlassian.net/123456789/attachments/att42 for
an attachment.
The file means "exists in Confluence as far as this job could see", not "was
indexed". A page is listed if the job saw it. An attachment is listed if
Confluence listed it on a page the job read, whether or not it was imported:
an attachment left out as too_large, images_not_requested or
attachment_cap is still in the file.
A document missing from the file is therefore not always a deleted page. Use the file to remove documents only when all of these hold:
-
The job finished with status
completed. A job that failed may leave no file, or an incomplete one. -
pages_failedis 0. A page that failed to fetch is listed, but its attachments are not. -
left_outhas none of these four reasons. After any of them, pages or attachments that still exist in Confluence are missing from the file:Reason What is missing from the file restrictedThe page and all its attachments. page_capEvery page past the limit, and their attachments. space_not_foundEverything in that space. attachments_unavailableThe page's attachments. The page itself is listed, and the failed counts stay 0. -
The comparison covers only documents from the same scope. A job for one space says nothing about pages in other spaces, and a job for one page tree says nothing about the rest of its space. Filter the collection's documents by
confluence_spacebefore comparing. For a job scoped to one page and the pages under it there is no stored key that names that tree, so the file cannot be used to find deleted pages in it. A page MOVED to another space is absent from its old space's file and still exists. Check its ID against the other spaces before deleting it. -
The token is the same one, with the same access, as earlier runs. A page the token can no longer see is absent from the file but still exists.
A document in that scope whose identity is not in the file is then either
gone from Confluence or moved out of the scope. Delete it with
DELETE /v2/collections/{name}/documents/{document_id} once it is confirmed
gone.
Billing
- Pages are stored as Markdown and bill as text: 0.5 credit per 3,000 characters of that Markdown, rounded up for each page. A page under 3,000 characters costs 0.5 credit.
- Attached pictures cost 0.5 credit each. Pictures, audio and video are
indexed, and billed, only when
include_imagesis true. - Other attachments bill at the rate for their file type and
processing_type, the same as any other indexed file. - Skipped pages and attachments cost nothing.
- A changed page is billed again as text. A page counts as changed when
its title, body, labels, comments or list of attachments changed, or when
include_commentsis set differently from the last import. See What counts as a change."include_comments": falseavoids re-billing pages whenever someone comments. - A page is indexed again, and billed again as text, in the import that
first links one of its media attachments (a picture, an audio file or a
video). That is how the attachment is linked to its page. It happens the
first time the attachment is indexed: when
include_imagesis turned on for pages imported without it, and also when a media attachment that an earlier run left out, at a limit or after a failed download, is picked up by a later run. A PDF, spreadsheet or text attachment picked up later does not re-index its page. - A media attachment that fails to index on every run can re-bill its
page. If a picture, audio file or video is downloaded but can never be
indexed (a damaged file, for example), its page is indexed again, and
billed again as text, each time
include_imagesis switched on or off. The same happens whenever its download result changes from one run to the next: a run where the download fails after one where it worked, or the other way round. Runs with the same result in a row cost nothing extra. Remove or replace the file in Confluence to stop it. - Audio and video attachments bill at their per-minute rates.
- Backup Protection + Storage: $0.44 per 1,000 credits stored per month. Each page and attachment stores the credits it cost to index.
- Queries are free. Reranking is billed at $0.05 per 1M tokens.
A worked example. A space has three pages of 27,263, 2,100 and 800 characters, and one of them has a picture attached:
| Item | How it is counted | Credits |
|---|---|---|
| 27,263-character page | 10 blocks of 3,000 characters, rounded up | 5.0 |
| 2,100-character page | 1 block | 0.5 |
| 800-character page | 1 block | 0.5 |
Attached picture, with include_images on | flat | 0.5 |
| First import | 6.5 |
Those 6.5 credits are also what the documents store, so storage for them is 6.5 × $0.44 / 1,000 per month. A second import with nothing changed costs 0 credits.
Troubleshooting
| Symptom | Cause and fix |
|---|---|
400 "Confluence did not accept this email and API token" | The token is wrong, expired or revoked, or it belongs to a different email. A scoped token missing read:space:confluence gets the same answer. |
400 "The API token can list Confluence spaces ... but cannot read pages" | A scoped token is missing read:page:confluence. Create a token with every scope listed in Step 2. |
400 "Confluence did not recognise this email and API token ... and answered as an anonymous visitor" | The site allows anonymous access and the token is wrong, so Confluence treated the request as a visitor. Check the email and the token. |
400 "No Confluence site was found" | No Atlassian site has that name. Check confluence_site. |
400 "The API token cannot see any Confluence spaces" | The account has no access to any space. Add it to the spaces to index. |
422 naming skip_existing or overwrite_existing | Remove the field. A second call already skips unchanged pages. Use force to re-index everything. |
422 on confluence_site | Only *.atlassian.net sites are accepted. |
409 confluence_import_running | Another Confluence import into the same collection has not finished. Wait for the job in job_id to finish, or cancel it. |
409 confluence_import_submitting | Another request to import into the same collection is being submitted right now. No job ID is given, because that request may still be refused. Retry in a few seconds. |
429 "Maximum concurrent indexing jobs reached" | The organization is at its limit of jobs running at once, across all sources. Wait for a job to finish. |
422 naming a field such as max_pages | The field is not accepted. The limits are fixed per job and cannot be set in the request. |
422 on confluence_email or confluence_api_token | Neither may contain spaces. The token must be plain ASCII, up to 2,048 characters. The email may be up to 320 characters. |
400 naming a metadata key such as source_role | Captain sets source_role and source_parent_identity itself. They can be read and filtered on. They cannot be set on an index request, and a metadata update cannot change or remove them. Sending one back unchanged on a metadata update is accepted. |
503 "could not be secured" or import_guard_unavailable | The credentials could not be encrypted, or Captain could not check whether another import is running. No job was started. Retry. |
503 plan_unverified | Captain could not read your plan, which sets the page limit. No job was started. Retry shortly. |
503 job_registry_unavailable | The job could not be recorded, so it was not started. Retry. |
503 job_start_unknown | The request to start the job got no answer, so the job may or may not be running. Check the job_id in the response with GET /v2/jobs/{job_id}. If it is not found after a few minutes, it did not start and the import can be submitted again. Send a new Idempotency-Key when you do: a request with the same key replays the same job ID and starts nothing. |
503 job_not_started | The job could not be started, and nothing is running. Retry. |
Job fails with an error_message that starts "Confluence import failed:" | The message says why and what to do: the token was refused or revoked, a scoped token is missing a required scope, the space or page was not found, the page is a draft, the page (or every listed page) could no longer be read because it was deleted or restricted after the import started, or Confluence kept rate limiting the import. Fix that and start a new job. Pages already indexed stay indexed. |
| Job fails with "an internal error on our side" | The fault is Captain's, not the token's. Start the job again. Unchanged pages are skipped. |
| Fewer pages than expected | Restricted pages the token cannot see are absent. Check source_report.left_out for a reached limit. |
409 confluence_credentials_required from POST /v2/jobs/{job_id}/reindex | A Confluence job cannot be retried from its job ID, because its credentials were encrypted for that one job. Start a new job. Unchanged pages are skipped. |
Security notes
- The email and token are encrypted before the job starts and never appear in plain text in job names, job status or logs.
- The encrypted token is kept in the job's execution record for up to 90 days. Only the indexing pipeline can decrypt it, and the encryption is bound to that job and your organization. Revoke the token after the import if it is not needed again.
- Captain only reads from Confluence. It never creates, edits or deletes pages.
- The token is sent to two places only: your site (
{site}.atlassian.net) and Atlassian's API gateway (api.atlassian.com), which scoped tokens require. - Revoke the token at any time at
id.atlassian.com.
Pages already indexed stay indexed. What a running job does depends on
how Confluence answers once the token is revoked. When Confluence refuses
the token outright, the job fails and says the token was refused. When it
answers each remaining page as forbidden or not found instead, those pages
cannot be told apart from pages the token was never allowed to see: they
are reported as
restrictedand the job completes. If every listed page is unreadable, the job fails. After revoking a token mid-run, check the job'sleft_outlist before trusting it as complete.