12 - FinOps: Cost Optimization for Agents and LLM Tokens

The problem: spending grows with iterations, not with requests

An application that calls a model once per request has a predictable cost: input tokens plus output tokens, times the number of requests. An agent breaks this proportion because inside every task there is a loop, and at every turn the model receives again everything that happened before. The cost of a task therefore depends on its length, and grows faster than the length would suggest. Controlling spending in an agent means acting on four levers: how much context is sent again, how much of that context can be reused at a reduced price, which model answers, and when to stop.

A warning about the numbers, valid for the whole article. Model prices change often and depend on the provider, and there is no price list here: the listings use prices invented for the example, declared as such in the code, and the results are the arithmetic of those parameters, not a measurement of a real service. Replace the parameters with those of your contract and run again. The listings are Python with the standard library only, run twice with identical output; no model was called and no paid call was made.

For the context of the series: the previous article showed how to collect metrics, including the tokens consumed; here we decide what to do with them. The next one puts the parts together in a constructed example.

Why a long task costs far more than a short one

The mechanism is an addition. At the first iteration the model receives the system prompt and the tool definitions (the prefix). At the second it receives the prefix plus what iteration one added: the model's reply and the tool's result. At the tenth it receives nine additions. If each iteration adds the same number of tokens, the total input tokens grow with the square of the number of turns, because the sum 1 + 2 + ... + n equals n(n+1)/2 and not n.

The listing computes the cost of a task of 1, 5, 10 and 20 iterations with a prefix of 2000 tokens, 700 tokens added at each turn and 300 generated (example values). The third column is the cost without cache; the fourth uses incremental caching, in which each iteration rereads at a reduced price what it had already sent and pays as a write only the new part. The multipliers 0.10 for reading and 1.25 for writing are those documented by Anthropic for the cache with a five-minute expiry; other providers have different rules.

# INVENTED prices for the example, in dollars per million tokens: not a price list.
INPUT_PER_M = 3.00
OUTPUT_PER_M = 15.00
CACHE_READ = 0.10   # multiplier documented by Anthropic for reads
CACHE_WRITE = 1.25  # multiplier for writes with a 5-minute expiry

PREFIX = 2000   # system prompt + tool definitions
STEP = 700      # tokens each iteration adds to the history
OUTPUT = 300    # tokens generated at each iteration


def dollars(tokens_in, tokens_out, multiplier=1.0):
    return tokens_in * INPUT_PER_M * multiplier / 1e6 + tokens_out * OUTPUT_PER_M / 1e6


def no_cache(n):
    total_in = sum(PREFIX + i * STEP for i in range(n))
    return total_in, dollars(total_in, n * OUTPUT)


def incremental_cache(n):
    """Each iteration rereads from the cache what it already had and writes only the new part."""
    cost = n * OUTPUT * OUTPUT_PER_M / 1e6
    already_cached = 0
    for i in range(n):
        input_tokens = PREFIX + i * STEP
        read = already_cached
        new = input_tokens - read
        cost += (read * CACHE_READ + new * CACHE_WRITE) * INPUT_PER_M / 1e6
        already_cached = input_tokens
    return cost


print(f"{'iter':>4} {'tokens in':>9} {'no cache':>12} {'with cache':>10} {'ratio vs 1 iter':>19}")
base = no_cache(1)[1]
for n in (1, 5, 10, 20):
    tin, c = no_cache(n)
    print(f"{n:>4} {tin:>9} {c:>11.4f}$ {incremental_cache(n):>9.4f}$ {c / base:>19.1f}")

# Output:
# iter tokens in     no cache with cache     ratio vs 1 iter
#    1      2000      0.0105$    0.0120$                 1.0
#    5     17000      0.0735$    0.0442$                 7.0
#   10     51500      0.1995$    0.0891$                19.0
#   20    173000      0.6090$    0.1947$                58.0

The last column is the lesson: twenty iterations do not cost twenty times a single one, they cost 58. The gap comes from the input tokens, which go from 2000 to 173,000, while the generated tokens grow only linearly. Note also the first row: with a single turn the cache costs more (0.0120 versus 0.0105), because you pay the write at 1.25 times and you never read anything. The cache pays off only if the prefix is reread before it expires.

The conditions under which prompt caching pays off are precise, and it is better to verify them in the provider's documentation than to trust a savings percentage:

• Identical prefix: the cache covers the prefix in the order in which the provider builds it (for Anthropic tools, then system, then messages), so changing an instruction at the start invalidates everything that follows.
• Reuse within the expiry: five minutes by default; a one-hour expiry exists, with a write at 2 times the input price.
• Minimum length: below a threshold that depends on the model (the documentation lists values from 512 to 4096 tokens) the request is processed without cache and without error.
• Checking usage: the cache_creation_input_tokens and cache_read_input_tokens fields of the response tell you whether the cache worked; if both are zero, it was not used.

The last item is the most useful in practice: a cache that does not kick in gives no errors, only a higher bill. For this reason the count of tokens read from the cache goes in the metrics next to the normal tokens, and the ratio between the two is the indicator to watch after every change to the system prompt.

A cap per task: the cost ledger

Measuring is not enough if the task keeps running while it spends. The remedy is a ledger that knows how much each task has already spent and refuses the call that would exceed the cap. The agent at that point does not fail: it returns what it has obtained so far, declared partial. The listing implements the ledger with a cap of 0.50 dollars per task (example value) and three scripted tasks.

from collections import defaultdict


class BudgetExhausted(Exception):
    pass


class Ledger:
    def __init__(self, cap_per_task):
        self.cap = cap_per_task
        self.per_task = defaultdict(float)
        self.per_agent = defaultdict(float)

    def charge(self, agent, task, dollars):
        if self.per_task[task] + dollars > self.cap:
            raise BudgetExhausted(f"{task}: {self.per_task[task]:.2f}$ spent, "
                                  f"the {dollars:.2f}$ call would exceed {self.cap:.2f}$")
        self.per_task[task] += dollars
        self.per_agent[agent] += dollars


def fake_agent(ledger, name, task, cost_per_step):
    results = []
    for n, cost in enumerate(cost_per_step, start=1):
        try:
            ledger.charge(name, task, cost)
        except BudgetExhausted as error:
            return {"status": "partial", "steps": results, "reason": str(error)}
        results.append(f"step {n}")
    return {"status": "complete", "steps": results}


led = Ledger(cap_per_task=0.50)
print(fake_agent(led, "research", "t1", [0.05, 0.08, 0.10]))
print(fake_agent(led, "research", "t2", [0.10, 0.20, 0.30, 0.40]))
print(fake_agent(led, "report", "t3", [0.02, 0.02]))
print({a: round(c, 2) for a, c in sorted(led.per_agent.items())})

# Output:
# {'status': 'complete', 'steps': ['step 1', 'step 2', 'step 3']}
# {'status': 'partial', 'steps': ['step 1', 'step 2'], 'reason': 't2: 0.30$ spent, the 0.30$ call would exceed 0.50$'}
# {'status': 'complete', 'steps': ['step 1', 'step 2']}
# {'report': 0.04, 'research': 0.53}

Task t2 stops at the third step because the 0.30 call would have brought the total to 0.60: the exception fires before charging, so the task's spending stays under the cap. The result returned is "partial" with the steps done and the reason, and it is up to the caller to decide whether to retry with more budget. The sum per agent (0.53 for research) is greater than the cap because it adds up two tasks, not one.

Two limits of the listing. It works in a single process: with several replicas the check and the charge must be a single atomic operation on a shared store, otherwise two simultaneous calls both pass the check. And the cost of a call is known only afterwards, from the tokens the provider declares in the response; the per-call cap (max_tokens) is the tool that limits the worst case before spending. Warning levels on the daily and monthly budget, with thresholds such as 50% or 80%, are a convention of yours, not a rule: choose the ones that give you time to react.

Choosing the model: the cascade and its weak point

Not all requests of an agent have the same difficulty, and large models cost many times the small ones. The literature has studied this for some time: the authors of FrugalGPT (arXiv:2305.05176) observe that the prices of paid models can differ by two orders of magnitude and describe three families of strategies, among them the model cascade, in which the cheap model is queried first and one moves to the expensive one only if the answer is not convincing. In their experiments, reported in the abstract, a learned cascade matches the best single model with a cost reduction of up to 98%, or surpasses it by 4 percentage points of accuracy at the same cost. It is a result of the authors, on their datasets and 2023 prices, not a promise.

Everything depends on how you decide that the small model's answer is not convincing. In the listing the decision is a confidence declared by the model, with a threshold. The 20 cases, the difficulties and the behavior of the small model are a script that I wrote: the small one answers well up to difficulty 2, is honest on the harder cases except one (difficulty 3, where it declares itself sure while being wrong) and the large one is always correct. The cost is in relative units (1 versus 15).

# 20 requests with a difficulty from 1 to 4 (data invented for the example, no seed needed)
REQUESTS = [1, 1, 1, 2, 1, 2, 1, 1, 3, 1, 2, 1, 1, 4, 1, 2, 3, 1, 1, 2]
COST = {"small": 1, "large": 15}   # relative units, not dollars


def small_model(difficulty):
    """Script: answers well up to difficulty 2; the stated confidence is almost always honest,
    but on case 3 it claims to be sure when it should not."""
    correct = difficulty <= 2
    confidence = 0.9 if difficulty <= 2 else (0.85 if difficulty == 3 else 0.3)
    return correct, confidence


def all_large():
    return len(REQUESTS) * COST["large"], 0


def cascade(threshold):
    cost = errors = 0
    for d in REQUESTS:
        correct, conf = small_model(d)
        cost += COST["small"]
        if conf >= threshold:
            errors += 0 if correct else 1   # the small model's answer is sent to the user
        else:
            cost += COST["large"]           # the large model answers, and the large one is always right
    return cost, errors


c0, e0 = all_large()
print(f"all on the large      cost {c0:>4}  errors {e0}")
for threshold in (0.95, 0.80, 0.50):
    c, e = cascade(threshold)
    print(f"cascade threshold {threshold:.2f}  cost {c:>4}  errors {e}  savings {100 * (1 - c / c0):.0f}%")

# Output:
# all on the large      cost  300  errors 0
# cascade threshold 0.95  cost  320  errors 0  savings -7%
# cascade threshold 0.80  cost   35  errors 2  savings 88%
# cascade threshold 0.50  cost   35  errors 2  savings 88%

The three rows tell three different things. With a threshold that is too high (0.95) the small model never convinces and the cascade costs more than always using the large one, because you pay for the useless first call. With 0.80 the cost falls by 88%, but two errors remain: they are the cases in which the model was wrong with confidence 0.85, and no threshold above or below catches them without throwing away the savings. The 88% figure is the arithmetic of the script, not a number to quote elsewhere.

The operational conclusion is that the savings of a cascade are not estimated, they are measured on your tasks, and must be read together with the errors it introduces. You therefore need a set of tasks with a known correct answer and a quality check: the article on evaluating agents covers this. Do not use it when an error costs far more than a call to the large model, or when you have no way of noticing the errors: in that case the savings are just a cost moved elsewhere. The percentages of a "70-20-10" split among simple, medium and difficult requests have no general source: your distribution comes from your own traffic.

Less context, same work: window, summary, retrieval

If the input tokens grow with the history, the most direct lever is not to send it all again. There are three techniques: keep only the last N exchanges (sliding window), replace the old part with a summary, or retrieve from memory only the fragments relevant to the current question. This third path is the one in the article on memory systems. The listing compares the first two with the full history over 30 exchanges of 120 tokens each (tokens and summarizer are fake: the summary is always 150 tokens).

TURNS = 30
TOKENS_PER_TURN = 120   # tokens (here: words) per user+reply exchange
THRESHOLD = 1000
WINDOW = 6
SUMMARY = 150           # length of the summary produced by the fake summarizer


def full():
    return sum(TOKENS_PER_TURN * t for t in range(TURNS))


def window():
    return sum(TOKENS_PER_TURN * min(t, WINDOW) for t in range(TURNS))


def with_summary():
    history = 0          # tokens of the raw history after the last summary
    summary = 0
    spent = 0
    summarizer_extra = 0
    for _ in range(TURNS):
        spent += summary + history
        history += TOKENS_PER_TURN
        if history > THRESHOLD:
            summarizer_extra += summary + history   # the summarizer reads everything
            summary, history = SUMMARY, 0
    return spent, summarizer_extra


s, extra = with_summary()
print("input tokens summed over 30 exchanges")
print(f"  full history        : {full()}")
print(f"  window of {WINDOW}         : {window()}")
print(f"  summary beyond {THRESHOLD}  : {s} (+ {extra} read by the summarizer)")

# Output:
# input tokens summed over 30 exchanges
#   full history        : 52200
#   window of 6         : 19080
#   summary beyond 1000  : 16470 (+ 3540 read by the summarizer)

The window reduces the input from 52,200 to 19,080 tokens, but it is blind: everything that left the window, including the user's instructions given at the start, no longer exists. The summary does better (16,470) but has a hidden cost, which the listing prints separately: the summarizer must read the history (another 3,540 tokens) and, if it is a model, the summary can omit or alter details. The trade-off is always the one between spending and fidelity, and the only way to say which loss is acceptable is to try it on your tasks.

Work that can wait: batch processing

Part of an agent's work has no user waiting: classifying an archive, enriching a dataset, regenerating descriptions, evaluating a test suite. For these jobs some providers offer cheaper asynchronous interfaces. OpenAI's documentation describes the Batch API as submitting groups of asynchronous requests with costs 50% lower, separate rate limits and a delivery time of 24 hours; Anthropic's describes message batches processed within 24 hours, many batches completing earlier, and results available for 29 days. Discounts change: check your provider's pricing page, and check whether they add up with the cache discount.

• Suitable: jobs without a user waiting, with high volumes and results that are read hours later (data enrichment, classification, nightly evaluations).
• Not suitable: conversations, interactive tools, any step of an agentic loop in which the next step depends on the response.
• To plan for: requests that expire without a result, to be retried, and the rule for how long you wait before declaring the job failed.

The common criterion is dependence on the result: an agentic loop needs the answer to decide the next step, so batch concerns the jobs around the agent, not its loop. Re-evaluating with a model a thousand answers of a test suite is the typical case in which batch pays off.

When an agent pays for itself, and when self-hosting makes sense

The final question is not technical: does the agent cost less than what it replaces? The calculation is simple and must be done with all the items, not just the calls to the model: the agent's monthly cost (model, hosting, amortized maintenance), the work saved (hours times hourly cost) and the initial development cost to recover. The second listing applies it to an invented case and also compares the fixed cost of a self-managed model with the pay-per-use one.

import math

# 1) ROI and break-even of an agent (INVENTED numbers for the example)
hours_per_week = 10
hourly_cost = 40
agent_cost_month = 600          # API + hosting + amortized maintenance
initial_development = 12_000
monthly_savings = hours_per_week * hourly_cost * 52 / 12
net = monthly_savings - agent_cost_month
print(f"gross savings per month: {monthly_savings:.0f}")
print(f"monthly ROI: {100 * net / agent_cost_month:.0f}%")
print(f"development break-even: {math.ceil(initial_development / net)} months")

# 2) Self-hosting versus API: from which volume it pays off (INVENTED prices)
api_price_per_m = 6.0           # dollars per million tokens (input+output average)
fixed_month = 1500              # server + electricity + the time of whoever runs it
break_even_volume_m = fixed_month / api_price_per_m
print(f"break-even: {break_even_volume_m:.0f} million tokens per month, "
      f"{break_even_volume_m / 30:.1f} per day")
for per_day in (1, 5, 10):
    api = per_day * 30 * api_price_per_m
    print(f"  {per_day:>2} M/day: API {api:>6.0f}$  fixed {fixed_month}$  -> {'self-hosting' if api > fixed_month else 'API'}")

# Output:
# gross savings per month: 1733
# monthly ROI: 189%
# development break-even: 11 months
# break-even: 250 million tokens per month, 8.3 per day
#    1 M/day: API    180$  fixed 1500$  -> API
#    5 M/day: API    900$  fixed 1500$  -> API
#   10 M/day: API   1800$  fixed 1500$  -> self-hosting

In the example the agent returns a monthly ROI of 189% but the development break-even arrives in 11 months: two ways of saying the same thing that sound very different, so report both. For self-hosting, the break-even with a fixed cost of 1,500 dollars a month and a pay-per-use price of 6 dollars per million is at 250 million tokens a month. The hard part is not the division but the list of fixed costs: renting or buying the machine, electricity, updates and the time of whoever looks after it. A smaller self-managed model, moreover, is not a neutral substitute for the large model: quality must be measured as for any cascade.

Frequently asked questions

What you can do now

You can explain why an agent's spending follows the length of the task and compute it with your own prices, recognize when the cache helps and when it costs more, put a cap per task that returns a partial result instead of an error, and evaluate a model cascade by looking at savings and errors together. You can reduce the context with a window or a summary knowing what is lost, reserve batch for jobs with no user waiting and report ROI and break-even as two distinct numbers. All the figures in the listings are examples: replace them with your own measurements.