Skip to main content
Version: v5.3

Tracking token usage

Overview​

Every conversation which is run by the AI Service costs money. To be precise, it costs tokens, which are monetary units charged for work done by the LLM providers, OpenAI or Anthropic. Previously, the amount of tokens used by each AI call was not recorded.

In Platform version 5.3, a new feature has been introduced which addresses this issue and allows you to keep a count of token usage for AI activity.

What exactly is being tracked?​

This new feature tracks token usage based on the concept of a turn. When a user sends a prompt or message to the AI Service, the AI Service answers it in one turn. A turn equates to one row of a chat history - holding the prompt, the answer, the summary, and the token usage total. The AI Service addresses this as /turns/{turnId}.

What the token tracking feature actually does​

The tracking token feature does the following:

  • Records token usage (input, output, total) for every turn, from real provider-reported figures, for both OpenAI and Anthropic.
  • Makes a turn's usage the combined total of every AI call it made — the tool loop, every agent in a supervisory team, the prompt-template router, both embedding calls and the asynchronous summarization call but never the last call's usage.
  • Keeps a conversation-level running total that stays consistent with the sum of its turns.
  • Presents usage in one provider-agnostic data structure: (_usage: { _input, _output, _total }). This is identical at turn and conversation level. In future platform versions, this could be extended with additional fields for cost or cached tokens.
  • Attributes each turn's total to the agent. This means embedding the call or summarization call that incurred it, to both the configured model and the model which the provider actually served.
  • Behaves identically across all entry points: streaming ((?sync=true&stream=true)), non-streaming synchronous ()(?sync=true)), and asynchronous (202/Kafka).
  • Reads older conversations and turns that predate this feature, as zeros, with no errors anywhere.
  • Works as something which is strictly additional to the existing process. No existing field, response structure, or behaviour changes, and no usage failure can fail a conversation.

API calls for tracking token usage​

To facilitate this new feature, four new API endpoints have been added to the existing conversation controller (aisvc/api/v1/conversations).

MethodPathPurposeAuthorization
GET/aisvc/api/v1/conversations/{id}/turns/{turnId}Returns the usage for one turn.
GET/aisvc/api/v1/conversations/{id}/turns/{turnId}/tokenusageReturn's one turn's total usage plus the sources behind it.conversation:{id} READ
GET/aisvc/api/v1/conversations/{id}/turnsReturns the conversation's specified turn's usage, paged.conversation:{id} READ
GET/aisvc/api/v1/conversations/{id}/tokenusage Returns the conversation usage total plus the sources behind it.conversation:{id} READ

Notes on the new API endpoints​

Authorization​

All four endpoints reuse the authorization rule already applied to GET /conversations/{id}. A turn and its cost are the conversation's own data, so neither needs a permission of its own.

Nesting under the conversation is deliberate as it makes the conversation the anchor for the authorization. The service additionally scopes every look-up to it, so a turn ID from another conversation is regarded as a 404 error rather than a readable turn.

Paging​

The API call GET .../turns takes the service's standard _offset and _pageSize (defaults 0 and 25) and returns the standard paged envelope, oldest turn first. The other three endpoints take no query parameters. A conversation's breakdown is one row per (source, model), however many turns it holds, so it does not page.

API examples​

Request usage for a turn​

To return usage for a turn, see example request below.

Request​

GET /aisvc/api/v1/conversations/{id}/turns/{turnId}

Response​

{
"_id": "3d4a17bc-2f8e-4c11-9a03-1c7b5f9a2d41",
"_conversationId": "b9c2f0e1-4c7a-4a2f-9a51-7f0b6a2f1c33",
"_teamId": "6f1b2c33-9d40-4e21-8a77-0c2e5b7d1a88",
"_prompt": "Which doors are on level 2?",
"_llmResponse": "There are 14 doors on level 2 …",
"_duration": 8231,
"_status": 1,
"_usage": { "_input": 4210, "_output": 512, "_total": 4722 },
"_metadata": {
"_createdAt": 1757500000000,
"_createdById": "auth0|…",
"_updatedAt": 1757500008231,
"_updatedById": "auth0|…"
}
}

Request usage for a conversation​

To return the usage total for a conversation plus the sources behind that usage, see the sample request below.

Request​


GET /aisvc/api/v1/conversations/{id}/tokenusage

Response​

{
"_conversationId": "b9c2f0e1-4c7a-4a2f-9a51-7f0b6a2f1c33",
"_usage": { "_input": 18432, "_output": 2310, "_total": 20742 },
"_sources": [
{
"_kind": "agent",
"_name": "data-model-specialist",
"_displayName": "Data Model Specialist",
"_model": "gpt-4o",
"_resolvedModel": "gpt-4o-2024-11-20",
"_provider": "openai",
"_usage": { "_input": 15980, "_output": 1902, "_total": 17882 },
"_calls": 14
},
{
"_kind": "agent",
"_name": "supervisor",
"_displayName": "Supervisor",
"_model": "claude-sonnet-4-6",
"_resolvedModel": "claude-sonnet-4-6",
"_provider": "anthropic",
"_usage": { "_input": 1902, "_output": 402, "_total": 2304 },
"_calls": 4
},
{
"_kind": "summarization",
"_name": "summary",
"_model": "gpt-4o",
"_resolvedModel": "gpt-4o-2024-11-20",
"_provider": "openai",
"_usage": { "_input": 402, "_output": 6, "_total": 408 },
"_calls": 2
},
{
"_kind": "embedding",
"_name": "response",
"_model": "text-embedding-3-large",
"_resolvedModel": "text-embedding-3-large",
"_provider": "openai",
"_usage": { "_input": 96, "_output": 0, "_total": 96 },
"_calls": 2
},
{
"_kind": "embedding",
"_name": "prompt",
"_model": "text-embedding-3-large",
"_resolvedModel": "text-embedding-3-large",
"_provider": "openai",
"_usage": { "_input": 52, "_output": 0, "_total": 52 },
"_calls": 2
}
]
}

Request for one turn's total plus the sources behind it​

To request the total usage for one specific turn's total (rather than the conversation) plus the sources behind it, see the request below.

Request​

GET /aisvc/api/v1/conversations/{id}/turns/{turnId}/tokenusage

Response​

This response is the same structure with _turnId set, and _usage being that turn's stored total rather than the conversation's total.

{
"_conversationId": "b9c2f0e1-4c7a-4a2f-9a51-7f0b6a2f1c33",
"_turnId": "3d4a17bc-2f8e-4c11-9a03-1c7b5f9a2d41",
"_usage": { "_input": 4210, "_output": 512, "_total": 4722 },
"_sources": [ "… as above, for this turn only" ]
}

Note: The field _turnId matches the path, the service signature, and the turn_id column it comes from. Nothing in a response carries the older chat vocabulary.

Server Side Events​

For an example of Server Side Event (SSE) token-usage (streaming callers), refer to the sample code below.

event: token-usage
data: {
"_message": { "_input": 4210, "_output": 512, "_total": 4722 },
"_conversation": { "_input": 18030, "_output": 2304, "_total": 20334 }
}

Notes: The settled running total, as returned by the increment statement,is emitted once per turn, after summary and before complete. _conversation, and is not a locally computed 'best guess'. It necessarily excludes the summarization top-up, which lands after the stream has closed. The event is documented as such, and a client wanting the final figure, re-reads the conversation.

Notes on using API​

Source kinds and names​

_kind_nameDescription
agentThe agent's user_typeEvery model call which that agent has made for this turn - includes tool-loop iterations, supervisory routing calls, and prompt-template router calls.
embeddingpromptEmbedding the user's prompt for history similarity search.
embeddingresponseEmbedding the prompt-and-answer pair for future retrieval.
summarizationsummaryThe asynchronous call that summarizes the turn for conversation memory.
unattributedunattributedSafety valve only — usage that reached a total without a source. Should never appear; logged at warn level if it does.

Note: _model is what the agent was configured with; _resolvedModel is what the provider reported serving, falling back to _model when nothing was reported. _calls counts the model calls that source made which reported usage — an agent that looped twenty times over a tool is one row with _calls: 20.

Error responses​

Error codeDescription
400Malformed ID
403Caller cannot read the conversation.
404Conversation does not exist or the turn does not belong to it.
200A conversation or turn that simply has no recorded usage returns 200 with zeros and an empty _sources.

Notes on the _usage property​

API callStructureNotes
GET /conversations (list)_usage per itemConversation running total
GET /conversations/{id}_usageConversation running total
GET /conversations/{id}?getChats=true (list)_usage per item in _chatsPer-turn totals; the history SELECT was widened to carry the usage columns across all of its strategies — firstN, lastN, similarity and all.
POST /conversations?sync=true_usage: { _message, _conversation } Non-streaming callers have no event to listen for, so the payload rides on the response.
POST /conversations (async 202)_usage: { _message, _conversation } on the stored resultThe asynchronous caller polls for the result and never sees a stream.
Kafka request/reply get_usageShares the same DTOs, so it is covered by the same change.

Javascript Platform API - relevant methods for conversations​

To interact with conversations, using the Javascript API, the AI Service (IafAISvc) currently exposes the following three methods:

  • createConversation
  • getConversations
  • streamConversation

For more information on the Javascript methods available, see IafAISvc.

Note: The _usage field, described earlier, arrives through those methods with no package change, because it is an additional field on responses which the platform already returns.