Tracking token usage
Overview
Every conversation which is run by the AI Service costs money. To be precise, it costs tokens, which are monetary units charged for work done by the LLM providers, OpenAI or Anthropic. Previously, the amount of tokens used by each AI call was not recorded.
In Platform version 5.3, a new feature has been introduced which addresses this issue and allows you to keep a count of token usage for AI activity.
What exactly is being tracked?
This new feature tracks token usage based on the concept of a turn. When a user sends a prompt or message to the AI Service, the AI Service answers it in one turn. A turn equates to one row of a chat history - holding the prompt, the answer, the summary, and the token usage total. The AI Service addresses this as /turns/{turnId}.
What the token tracking feature actually does
The tracking token feature does the following:
- Records token usage (input, output, total) for every turn, from real provider-reported figures, for both OpenAI and Anthropic.
- Makes a turn's usage the combined total of every AI call it made — the tool loop, every agent in a supervisory team, the prompt-template router, both embedding calls and the asynchronous summarization call but never the last call's usage.
- Keeps a conversation-level running total that stays consistent with the sum of its turns.
- Presents usage in one provider-agnostic data structure:
(_usage: { _input, _output, _total }). This is identical at turn and conversation level. In future platform versions, this could be extended with additional fields for cost or cached tokens. - Attributes each turn's total to the agent. This means embedding the call or summarization call that incurred it, to both the configured model and the model which the provider actually served.
- Behaves identically across all entry points: streaming (
(?sync=true&stream=true)), non-streaming synchronous ()(?sync=true)), and asynchronous (202/Kafka). - Reads older conversations and turns that predate this feature, as zeros, with no errors anywhere.
- Works as something which is strictly additional to the existing process. No existing field, response structure, or behaviour changes, and no usage failure can fail a conversation.
API calls for tracking token usage
To facilitate this new feature, four new API endpoints have been added to the existing conversation controller (aisvc/api/v1/conversations).
| Method | Path | Purpose | Authorization |
|---|---|---|---|
| GET | /aisvc/api/v1/conversations/{id}/turns/{turnId} | Returns the usage for one turn. | |
| GET | /aisvc/api/v1/conversations/{id}/turns/{turnId}/tokenusage | Return's one turn's total usage plus the sources behind it. | conversation:{id} READ |
| GET | /aisvc/api/v1/conversations/{id}/turns | Returns the conversation's specified turn's usage, paged. | conversation:{id} READ |
| GET | /aisvc/api/v1/conversations/{id}/tokenusage | Returns the conversation usage total plus the sources behind it. | conversation:{id} READ |
Notes on the new API endpoints
Authorization
All four endpoints reuse the authorization rule already applied to GET /conversations/{id}. A turn and its cost are the conversation's own data, so neither needs a permission of its own.
Nesting under the conversation is deliberate as it makes the conversation the anchor for the authorization. The service additionally scopes every look-up to it, so a turn ID from another conversation is regarded as a 404 error rather than a readable turn.
Paging
The API call GET .../turns takes the service's standard _offset and _pageSize (defaults 0 and 25) and returns the standard paged envelope, oldest turn first. The other three endpoints take no query parameters. A conversation's breakdown is one row per (source, model), however many turns it holds, so it does not page.
API examples
Request usage for a turn
To return usage for a turn, see example request below.
Request
GET /aisvc/api/v1/conversations/{id}/turns/{turnId}
Response
{
"_id": "3d4a17bc-2f8e-4c11-9a03-1c7b5f9a2d41",
"_conversationId": "b9c2f0e1-4c7a-4a2f-9a51-7f0b6a2f1c33",
"_teamId": "6f1b2c33-9d40-4e21-8a77-0c2e5b7d1a88",
"_prompt": "Which doors are on level 2?",
"_llmResponse": "There are 14 doors on level 2 …",
"_duration": 8231,
"_status": 1,
"_usage": { "_input": 4210, "_output": 512, "_total": 4722 },
"_metadata": {
"_createdAt": 1757500000000,
"_createdById": "auth0|…",
"_updatedAt": 1757500008231,
"_updatedById": "auth0|…"
}
}
Request usage for a conversation
To return the usage total for a conversation plus the sources behind that usage, see the sample request below.
Request
GET /aisvc/api/v1/conversations/{id}/tokenusage
Response
{
"_conversationId": "b9c2f0e1-4c7a-4a2f-9a51-7f0b6a2f1c33",
"_usage": { "_input": 18432, "_output": 2310, "_total": 20742 },
"_sources": [
{
"_kind": "agent",
"_name": "data-model-specialist",
"_displayName": "Data Model Specialist",
"_model": "gpt-4o",
"_resolvedModel": "gpt-4o-2024-11-20",
"_provider": "openai",
"_usage": { "_input": 15980, "_output": 1902, "_total": 17882 },
"_calls": 14
},
{
"_kind": "agent",
"_name": "supervisor",
"_displayName": "Supervisor",
"_model": "claude-sonnet-4-6",
"_resolvedModel": "claude-sonnet-4-6",
"_provider": "anthropic",
"_usage": { "_input": 1902, "_output": 402, "_total": 2304 },
"_calls": 4
},
{
"_kind": "summarization",
"_name": "summary",
"_model": "gpt-4o",
"_resolvedModel": "gpt-4o-2024-11-20",
"_provider": "openai",
"_usage": { "_input": 402, "_output": 6, "_total": 408 },
"_calls": 2
},
{
"_kind": "embedding",
"_name": "response",
"_model": "text-embedding-3-large",
"_resolvedModel": "text-embedding-3-large",
"_provider": "openai",
"_usage": { "_input": 96, "_output": 0, "_total": 96 },
"_calls": 2
},
{
"_kind": "embedding",
"_name": "prompt",
"_model": "text-embedding-3-large",
"_resolvedModel": "text-embedding-3-large",
"_provider": "openai",
"_usage": { "_input": 52, "_output": 0, "_total": 52 },
"_calls": 2
}
]
}
Request for one turn's total plus the sources behind it
To request the total usage for one specific turn's total (rather than the conversation) plus the sources behind it, see the request below.
Request
GET /aisvc/api/v1/conversations/{id}/turns/{turnId}/tokenusage
Response
This response is the same structure with _turnId set, and _usage being that turn's stored total rather than the conversation's total.
{
"_conversationId": "b9c2f0e1-4c7a-4a2f-9a51-7f0b6a2f1c33",
"_turnId": "3d4a17bc-2f8e-4c11-9a03-1c7b5f9a2d41",
"_usage": { "_input": 4210, "_output": 512, "_total": 4722 },
"_sources": [ "… as above, for this turn only" ]
}
Note: The field
_turnIdmatches the path, the service signature, and theturn_idcolumn it comes from. Nothing in a response carries the older chat vocabulary.
Server Side Events
For an example of Server Side Event (SSE) token-usage (streaming callers), refer to the sample code below.
event: token-usage
data: {
"_message": { "_input": 4210, "_output": 512, "_total": 4722 },
"_conversation": { "_input": 18030, "_output": 2304, "_total": 20334 }
}
Notes: The settled running total, as returned by the increment statement,is emitted once per turn, after
summaryand beforecomplete. _conversation, and is not a locally computed 'best guess'. It necessarily excludes the summarization top-up, which lands after the stream has closed. The event is documented as such, and a client wanting the final figure, re-reads the conversation.
Notes on using API
Source kinds and names
| _kind | _name | Description |
|---|---|---|
agent | The agent's user_type | Every model call which that agent has made for this turn - includes tool-loop iterations, supervisory routing calls, and prompt-template router calls. |
embedding | prompt | Embedding the user's prompt for history similarity search. |
embedding | response | Embedding the prompt-and-answer pair for future retrieval. |
summarization | summary | The asynchronous call that summarizes the turn for conversation memory. |
unattributed | unattributed | Safety valve only — usage that reached a total without a source. Should never appear; logged at warn level if it does. |
Note:
_modelis what the agent was configured with;_resolvedModelis what the provider reported serving, falling back to_modelwhen nothing was reported._callscounts the model calls that source made which reported usage — an agent that looped twenty times over a tool is one row with_calls: 20.
Error responses
| Error code | Description |
|---|---|
400 | Malformed ID |
403 | Caller cannot read the conversation. |
404 | Conversation does not exist or the turn does not belong to it. |
200 | A conversation or turn that simply has no recorded usage returns 200 with zeros and an empty _sources. |
Notes on the _usage property
| API call | Structure | Notes |
|---|---|---|
GET /conversations (list) | _usage per item | Conversation running total |
GET /conversations/{id} | _usage | Conversation running total |
GET /conversations/{id}?getChats=true (list) | _usage per item in _chats | Per-turn totals; the history SELECT was widened to carry the usage columns across all of its strategies — firstN, lastN, similarity and all. |
POST /conversations?sync=true | _usage: { _message, _conversation } | Non-streaming callers have no event to listen for, so the payload rides on the response. |
POST /conversations (async 202) | _usage: { _message, _conversation } on the stored result | The asynchronous caller polls for the result and never sees a stream. |
| Kafka request/reply get | _usage | Shares the same DTOs, so it is covered by the same change. |
Javascript Platform API - relevant methods for conversations
To interact with conversations, using the Javascript API, the AI Service (IafAISvc) currently exposes the following three methods:
createConversationgetConversationsstreamConversation
For more information on the Javascript methods available, see IafAISvc.
Note: The
_usagefield, described earlier, arrives through those methods with no package change, because it is an additional field on responses which the platform already returns.