Is it cheaper to do it in-house? It depends on four numbers
"Self-hosted costs less than the cloud" is a sentence repeated without numbers. The honest answer is that it depends on four things that only you can measure: how many tokens a request consumes, how much the vendor charges you per million, how much the machine costs per month, and how many requests that machine can actually serve. Anyone can read the first three; the fourth gets forgotten, and it changes the result. In this article you will not find a price list: you will find a runnable calculation model, with assumptions declared as such, to rerun with your own prices on the day you decide.
A clarification about the content. Vendor and server prices change, and an article that reports them goes stale in weeks: that is why the values in the program are marked as assumptions and no euro figure in the example is a price list. The quantities that can instead be derived from primary sources, such as a model's memory or Ollama's defaults, are reported with their source.
Why an agent consumes more tokens than a chatbot
An agent does not make one call, it makes one for every turn, and at every turn it sends the whole history back to the model: system instructions, messages, tool results. If the instructions are 800 tokens and each turn adds 350 (150 for the message, 200 for the reply), at the sixth turn the input request is 2,550 tokens and the sum over the six turns exceeds ten thousand, against the 1,200 generated. The input grows like an arithmetic sum, not like a multiple: that is why a loop with many rounds costs far more than the length of the final answer suggests.
The count changes if the vendor or the engine reuses the already-processed initial part of the prompt. Ollama's API distinguishes, in its response, the prompt tokens evaluated from scratch (prompt_eval_duration concerns the "not cached" tokens); and Mastra's documentation warns that, once the window's message limit is exceeded, every turn invalidates the vendor's prompt cache. The model below does not account for these discounts: it considers the worst case, the one in which all the input is paid for.
The cost model, runnable
The program has an ASSUMPTIONS table with six assumptions (price per million tokens in and out, monthly cost of the machine, hours of management, the value of an hour, the machine's tokens per second) and four functions: the tokens of a request, the cloud cost, the cost of your own machine and the capacity. Every value in the table was invented by me for the calculation, it is not a price list: replace them with yours. The output is identical on every run.
"""Cost model: cloud agents (per token) versus a machine of your own.
Every number in ASSUMPTIONS is an ASSUMPTION to replace with yours: it is not a price list."""
ASSUMPTIONS = {
"price_in_eur_per_million": 0.50, # assumption: prompt tokens
"price_out_eur_per_million": 1.50, # assumption: generated tokens
"machine_eur_month": 40.0, # assumption: server dedicated to inference
"management_hours_month": 3.0, # assumption: updates, monitoring, incidents
"hourly_cost_eur": 50.0, # assumption: the value of your time
"machine_tokens_s": 10.0, # assumption: generation, one stream at a time
}
SECONDS_PER_MONTH = 30 * 24 * 3600
def tokens_per_request(system=800, message=150, turns=6, reply=200):
"""An agent resends the whole history at every turn: the input grows like a staircase."""
tokens_in = sum(system + k * (message + reply) for k in range(turns))
return tokens_in, turns * reply
def cloud_cost(requests, tok_in, tok_out, s=ASSUMPTIONS):
return requests * (tok_in * s["price_in_eur_per_million"]
+ tok_out * s["price_out_eur_per_million"]) / 1e6
def own_cost(machines=1, s=ASSUMPTIONS):
return machines * s["machine_eur_month"] + s["management_hours_month"] * s["hourly_cost_eur"]
def request_capacity(tok_out, utilization=0.5, s=ASSUMPTIONS):
"""Physical limit: how many requests the machine generates, counting only output tokens."""
return int(s["machine_tokens_s"] * SECONDS_PER_MONTH * utilization // tok_out)
def machines_needed(requests, tok_out):
cap = request_capacity(tok_out)
return -(-requests // cap) # ceiling division: half a machine does not exist
tok_in, tok_out = tokens_per_request()
cap = request_capacity(tok_out)
per_request = cloud_cost(1, tok_in, tok_out)
print(f"one request: {tok_in} input tokens, {tok_out} output tokens (6 turns)")
print(f"cloud: {per_request:.6f} EUR per request")
print(f"one machine handles {cap:,} requests/month at 50% utilization")
print(f"fixed cost with one machine: {own_cost(1):.2f} EUR/month")
print("\nrequests/month | cloud EUR | machines | own EUR | cheaper")
for r in (1_000, 10_000, 50_000, 100_000, 200_000, 400_000):
n = machines_needed(r, tok_out)
c, p = cloud_cost(r, tok_in, tok_out), own_cost(n)
print(f"{r:>14,} | {c:>9.2f} | {n:>8} | {p:>7.2f} | {'cloud' if c < p else 'own'}")
def first_breakeven(step=100):
"""First level of requests/month at which your own machine costs less than the cloud."""
r = step
while cloud_cost(r, tok_in, tok_out) <= own_cost(machines_needed(r, tok_out)):
r += step
return r
print(f"\nnaive break-even (a single machine, capacity ignored): {own_cost(1) / per_request:,.0f}")
print(f"break-even with capacity (machines added when needed): {first_breakeven():,}")
def tokens_per_second(reply):
"""Ollama documentation formula: eval_count / eval_duration * 10^9."""
return reply["eval_count"] / reply["eval_duration"] * 1e9
# Example response taken from the Ollama API documentation: NOT a measurement of mine.
example = {"prompt_eval_count": 26, "prompt_eval_duration": 130079000,
"eval_count": 259, "eval_duration": 4232710000}
print(f"\nexample from the documentation: {tokens_per_second(example):.1f} tokens/s in generation")
# Output:
# one request: 10050 input tokens, 1200 output tokens (6 turns)
# cloud: 0.006825 EUR per request
# one machine handles 10,800 requests/month at 50% utilization
# fixed cost with one machine: 190.00 EUR/month
#
# requests/month | cloud EUR | machines | own EUR | cheaper
# 1,000 | 6.83 | 1 | 190.00 | cloud
# 10,000 | 68.25 | 1 | 190.00 | cloud
# 50,000 | 341.25 | 5 | 350.00 | cloud
# 100,000 | 682.50 | 10 | 550.00 | own
# 200,000 | 1365.00 | 19 | 910.00 | own
# 400,000 | 2730.00 | 38 | 1670.00 | own
#
# naive break-even (a single machine, capacity ignored): 27,839
# break-even with capacity (machines added when needed): 51,300
#
# example from the documentation: 61.2 tokens/s in generation
The function that makes the difference is request_capacity: a machine that generates 10 tokens per second, used at 50%, produces about 13 million output tokens in a month, that is 10,800 requests of 1,200 tokens. Without that limit the count would be naive: 190 euro fixed divided by 0.006825 euro per request gives a break-even of 27,839 requests per month, but a single machine handles 10,800. The last line says it: accounting for the machines to be added, the break-even moves to 51,300 requests per month, almost double.
Read the table by tiers. Up to 10,000 requests the cloud costs a fraction of the machine's fixed cost; at 50,000 the two costs nearly coincide (341.25 against 350); from 100,000 up, your own machine wins. The conclusion holds for these assumptions and for no others, but the shape is robust: the cloud cost grows linearly with requests, your own grows in steps, and the first crossing point is always further out than the count without capacity says. Change the tokens per second and you will see everything shift.
The number most worth measuring is exactly that one. Ollama's API reports for every response eval_count and eval_duration (in nanoseconds), and the documentation gives the formula for tokens per second: eval_count divided by eval_duration times 109. The last line of the program applies it to the documentation's example response, 259 tokens in 4.23 seconds, which give 61.2 tokens per second: it is a documentation example, not the speed of a machine of yours, and certainly not the 10 of the assumption. Replace the number with the one you measure on your hardware and your model.
Memory: why the machine handles less than you think
The bottleneck is not only speed, it is memory. A model takes up memory for the weights and, on top of that, for the key-value cache (KV cache), which grows with the context and with the number of requests served together. The per-token cache is computed from the model's configuration: two tensors (keys and values) for every layer, for every key-value attention head, times the head dimension, times the bytes of each value. The program uses the published configuration of an example model, Qwen2.5-7B-Instruct on Hugging Face, because a concrete case is needed: it is an example and not a model recommendation.
# Values read from config.json of Qwen/Qwen2.5-7B-Instruct on Hugging Face (2026-11-13)
HIDDEN, HEADS, LAYERS, KV_HEADS = 3584, 28, 28, 4
PARAMETERS = 7_615_616_512 # total declared by the model's site
head_dim = HIDDEN // HEADS
def kv_cache_bytes(context, bytes_per_value=2, parallel=1):
# 2 = keys and values; f16 = 2 bytes (Ollama default for the K/V cache)
return 2 * LAYERS * KV_HEADS * head_dim * bytes_per_value * context * parallel
GIB = 1024 ** 3
print("head_dim:", head_dim)
print("cache per token:", kv_cache_bytes(1), "bytes")
for ctx, par in [(4096, 1), (32768, 1), (8192, 4), (32768, 4)]:
print(f"context {ctx:>6} x {par} parallel requests: {kv_cache_bytes(ctx, parallel=par) / GIB:5.2f} GiB")
for bit in (16, 8, 4):
print(f"weights at {bit:>2} nominal bits: {PARAMETERS * bit / 8 / GIB:5.2f} GiB (without overhead)")
# Output:
# head_dim: 128
# cache per token: 57344 bytes
# context 4096 x 1 parallel requests: 0.22 GiB
# context 32768 x 1 parallel requests: 1.75 GiB
# context 8192 x 4 parallel requests: 1.75 GiB
# context 32768 x 4 parallel requests: 7.00 GiB
# weights at 16 nominal bits: 14.19 GiB (without overhead)
# weights at 8 nominal bits: 7.09 GiB (without overhead)
# weights at 4 nominal bits: 3.55 GiB (without overhead)
The constants at the top come from the model's config.json: 28 layers, 28 heads, 4 key-value heads (the technique called GQA, which reduces the cache) and hidden size 3584, hence head dimension 3584 divided by 28, that is 128. The result: 57,344 bytes, 56 KiB, for every context token. With a context of 32,768 tokens the cache of a single request weighs 1.75 GiB; with 4 parallel requests, 7 GiB. The weights at 4 nominal bits are worth 3.55 GiB, but that is a lower bound without overhead: you find the real file on the model's page. The total has a single culprit, the long context multiplied by the parallel requests.
Ollama exposes exactly these levers, with defaults that its FAQ declares:
OLLAMA_CONTEXT_LENGTH: the context window, 4096 tokens if you do not set it;OLLAMA_NUM_PARALLEL: the requests processed together by each model, 1 by default; the RAM required scales with this value times the context length;OLLAMA_MAX_QUEUE: the requests queued before rejecting the next ones, 512 by default (beyond that, it answers 503);OLLAMA_KEEP_ALIVE: how long a model stays in memory after the last request, 5 minutes by default;OLLAMA_KV_CACHE_TYPE: the cache precision,f16by default;q8_0uses about half the memory.
All five levers shift cost and behavior. Raising parallelism multiplies the cache memory, as the program shows. The 512-slot queue turns a peak into latency instead of an error, up to a point. A KEEP_ALIVE of five minutes means that the first user after a pause pays for loading the model. And the quantized cache reduces memory at the expense of precision: the same FAQ warns that models with many heads per group, like the Qwen2 family, may suffer more. There is no right value: there is the one you measured on your load.
OLLAMA_CONTEXT_LENGTH=8192 ollama serve # context per request
OLLAMA_NUM_PARALLEL=2 # parallel requests per model
OLLAMA_MAX_QUEUE=64 # shorter queue: reject early instead of piling up
OLLAMA_KEEP_ALIVE=30m # model kept warm for half an hour
The variables are those of Ollama's FAQ; the values are examples to adapt and were not run, because starting the service opens a port. A security note from the same FAQ: Ollama binds to 127.0.0.1:11434 and changing it with OLLAMA_HOST exposes it to the network, so if more than one machine must reach it, put your own authentication in front of it: I did not verify that it offers one of its own.
What the model does not count
The cost model is deliberately small, and its limits are the honest part of the analysis. It does not count:
- the time to evaluate the prompt: on CPU it can exceed the generation time, and the program ignores it, so the real capacity is lower than estimated;
- quality: a small model and a large one do not answer the same way, and redoing failed requests has a cost that does not appear;
- the reliability of a single machine: maintenance, updates, a failure at night;
- data: cost is not the only reason to keep requests in-house, and for some workloads it is the decisive one;
- the maintenance of the agent's software, which changes with every version, as the earlier articles in the series show.
The first point explains why the outcome should be read as optimistic for your own machine: if the real capacity is lower, the machines needed grow and the break-even moves even further out. The other four have no figure, and that does not make them secondary: they are the reasons why the choice is rarely made on cost alone. If you notice that the decision depends on one of them, the model is no longer enough: you need a trial on your own load, for a week, with your own data.
How to use the results
The procedure that comes out of it is short. First, record for a week the input and output tokens of every request, which vendors normally report in the response (Ollama does so with prompt_eval_count and eval_count). Second, measure your hardware's tokens per second with the fields of Ollama's response. Third, put that day's prices into the ASSUMPTIONS table and rerun the program. Fourth, compare the cost with the value of what you move in-house: privacy, independence from the vendor, response times. A hybrid architecture, with simple requests local and difficult ones with a vendor, is an option to evaluate after the measurement, not before.
What you can do now
You can break the cost of an agent down into tokens, machine and management time, you know why tokens grow at every turn and why capacity shifts the break-even, and you can compute a model's memory from its configuration. You have a program you can rerun with your own prices. For what the agent remembers between one session and the next, and how much it weighs in the prompt, there is the article on Letta; for tools and workflows in TypeScript, the one on Mastra. Before spending, look at the comparison between the frameworks, because the framework decides how many rounds your agent makes; afterwards, the article on observability shows you how to measure the rounds. For the details of Ollama in production there is also the article on Ollama setup.
Frequently asked questions
Does a self-hosted model cost less than the cloud?
It depends on four numbers: tokens per request, the cloud price per million tokens, the machine's monthly cost and the requests the machine can serve. With the article's assumptions the break-even falls at 51,300 requests per month, but they are assumptions: rerun the program with your prices of the day.
How much memory does it take to run a 7-billion-parameter model?
For the weights, parameters times bits divided by eight: at 4 nominal bits a 7.6-billion model takes about 3.5 GiB, to which is added the key-value cache, 56 KiB per context token in the example model. The model's real file and the engine's overhead change the total, so measure it.
Can I trust the example's prices and tokens per second?
No: they are assumptions declared to make the calculation run. The only figures taken from sources are those of the model's config.json and Ollama's defaults, read on 13 November 2026. Prices and speeds must be replaced with what you read and measure yourself.
Sources
- Hugging Face: config.json of Qwen2.5-7B-Instruct - layers, attention heads, key-value heads and hidden size for the cache calculation, consulted on 2026-11-13
- Hugging Face: Qwen2.5-7B-Instruct model card - Apache-2.0 license and total parameters (7,615,616,512) from the site's API, consulted on 2026-11-13
- Ollama: FAQ - 4096-token context, NUM_PARALLEL, MAX_QUEUE, KEEP_ALIVE, KV_CACHE_TYPE, listening address, consulted on 2026-11-13
- Ollama: API documentation - the eval_count, eval_duration, prompt_eval_duration fields and the tokens-per-second formula, consulted on 2026-11-13
- Mastra docs: Memory class - the sliding window invalidates the vendor's prompt cache, consulted on 2026-11-13
The euro values and the tokens per second in the cost program are the author's assumptions and come from no source; the program was run twice with python3 -I and the same output. The memory program uses the constants of the config.json listed, read on 2026-11-13.

Commenti
Caricamento commenti...
Sign in to comment