Cost and tokens

Cost per task is a release metric

Teams usually meet agent cost as a surprise on an invoice. We treat it as a number you design for, measure per task and check before every release. The number that matters to the business is cost per task: what it costs to finish one unit of work, such as one handled request or one processed document.

In this lesson we find where tokens go, measure them, and pull the levers in an order that protects quality. We do not discuss provider rate cards. They change, and your agreement may differ. Use the numbers from your own contract.

Where the tokens go

One agent request carries much more than the question. Each part has its own driver.

An agent calls the model many times per task, and each call replays the history and the tool definitions. That is why a "cheap" model with a bloated loop can cost more than an "expensive" model with a tight one. The loop and the context decide the bill at least as much as the model does.

Our post on what drives cost and latency in inference on Azure Databricks explains the serving options in more depth: batch, pay-per-token, provisioned throughput and self-hosted.

Step 1: measure per task, per step

You cannot lower what you have not measured. Capture, for every task:

• Tokens in and out per model call, and per step in the loop.
• Latency per step and per task.
• The number of loop passes and retries.
• A quality score from the evaluation set, next to the cost.

MLflow Tracing records each step of an agent run with its inputs, outputs, latency and token usage, so the data exists once tracing is on. Unity Gateway can log requests and responses to Delta tables and track usage, including token usage and latency, in system tables, with cost analysis by service, model, principal and tag.

# Cost per task from token counts. Rates come from YOUR agreement, per million tokens.
RATE_IN_PER_M  = 1.0   # placeholder: replace with your contract rate for input tokens
RATE_OUT_PER_M = 4.0   # placeholder: replace with your contract rate for output tokens
 
def call_cost(tokens_in: int, tokens_out: int) -> float:
    return tokens_in / 1_000_000 * RATE_IN_PER_M + tokens_out / 1_000_000 * RATE_OUT_PER_M
 
def task_cost(calls: list[dict]) -> float:
    """calls: one dict per model call in the task, with tokens_in and tokens_out."""
    return sum(call_cost(c["tokens_in"], c["tokens_out"]) for c in calls)
 
def heaviest_step(calls: list[dict]) -> dict:
    return max(calls, key=lambda c: call_cost(c["tokens_in"], c["tokens_out"]))

The rates are placeholders on purpose. Keep the money figures in configuration, and keep the token counts as the primary measure, since tokens do not change when a provider changes its rates.

Step 2: pull the levers, in this order

Pick levers by what the measurements show. Test each one on your evaluation set before it ships.

The first rows are the cheapest to pull and carry the least quality risk. Routing deserves a warning: it is a quality control problem, not a cost switch. A router that sends hard cases to a weak model saves tokens and loses answers. Our post on model routing as a quality control problem goes deeper.

Step 3: guardrails on spend

Put limits where an agent cannot talk its way past them.

• Per-task limits in the harness: steps, time and tokens, as in lesson 1.
• Per-team and per-user limits at the gateway. Unity Gateway supports rate limits on model and MCP services, and budgets with per-user thresholds and hard caps.
• Alerts on spend per use case, so an agent that starts looping shows up the same day.

Step 4: make cost per task a release gate

Add a rule to the pipeline from lesson 4: a new version may ship only if quality holds on the evaluation set and cost per task stays within the agreed budget. A prompt change that improves quality by a point and doubles the tokens should be a conscious decision, not an accident found at month-end.

Record the result per release, next to the quality score. Over a few releases you will see which changes paid off.

Common mistakes

• Optimising the model first. Fix the loop and the context first.
• Averages only. Look at the expensive tail: the tasks that loop.
• Cutting quality for cost without measuring. Every lever needs the evaluation set.
• Hard-coding rates. Keep them in configuration.

Summary

Measure tokens per step and cost per task. Pull the cheap, low-risk levers first. Enforce limits at the harness and the gateway. Gate releases on quality and cost together.

Next lesson: how to bring your own model and swap models without a rebuild.