14 - AI On-Prem vs Cloud: Privacy, Cost, and Data Sovereignty

For most business workloads, cloud AI is a privacy tax you do not need to pay. This is not an ideological stance: it is what you find when you line up the real marginal cost of an on-prem model against a metered API's list price, row by row.

Method note: the infrastructure numbers cited here (VPS, monthly cost) are the ones from my own production stack, where until 23 October 2026 on-prem Ollama was the AI provider for the RAG engine and for the dev-tools: not a hypothetical sales-deck example.

Why most companies do not need the most powerful model on the market

The question almost no decision-maker asks before signing a metered AI contract is: does my workload actually need the most capable model available, or am I paying for a capacity ceiling I never touch? Internal document search, ticket classification, email summarization, structured extraction from forms: for many of these tasks a small local model may be enough, but "may be enough" is a hypothesis to verify on your own documents, not a fact: you measure it by comparing the answers of two models on a sample of your own cases. The rest (complex multi-step reasoning, sophisticated creative generation, advanced multimodal tasks) is where cloud keeps a real, not imagined, advantage.

The problem is not picking a model that is too weak. It is paying for a fee sized for the worst case while most of the requests are probably the average case. Which share that is, you only know by counting it on your own traffic.

The real cost: what the VPS actually costs, not the list price

My production stack (Spring Boot backend, Angular SSR frontend, an ML microservice, PostgreSQL database, Redis cache, and Ollama as the inference engine) runs entirely on a single Hetzner CCX23 VPS (4 dedicated vCPU, 16 GB RAM), shared across every one of those services. The cost of that VPS is the entire infrastructure spend line: there is no separate second invoice for AI, because Ollama runs on the same machine, on the same cores, with no metered third-party provider to pay.

The calculation you can redo: from fixed cost to break-even point

"Zero marginal cost" is true for the monthly invoice, but the server is paid for anyway: the honest question is how many tokens you must generate for that fixed cost to become lower than a metered invoice, and whether your hardware can produce them. I do not quote list prices because they change fast: an analysis by Andreessen Horowitz estimates that for a model of equivalent performance the cost falls by about ten times a year, with the reservation, stated by the authors themselves, that the time scale may change. That is why the calculation takes prices as parameters.

The script starts from three assumptions, all to be replaced with your own: the monthly cost of a server dedicated to inference (40 euros), the number of tokens per second your hardware generates with your model (10, to be measured) and the share of the month in which it actually works (20%). From them it derives the monthly capacity and, for three hypothetical prices of a metered API, the break-even volume, that is, the millions of tokens per month beyond which the server costs less.

# From a server's fixed cost to the break-even point against a pay-per-use API.
# ALL values are ASSUMPTIONS to replace with your own (the day's price list, a measurement of your hardware).
SECONDI_MESE = 30 * 24 * 3600

costo_server_mese = 40.0      # EUR per month, server dedicated to inference
token_al_secondo = 10.0       # throughput measured on YOUR hardware with YOUR model
utilizzo = 0.20               # share of the month in which the server actually generates

capacita_100 = token_al_secondo * SECONDI_MESE
capacita = capacita_100 * utilizzo
print(f"capacity at full load: {capacita_100 / 1e6:5.1f} million tokens/month")
print(f"capacity at {utilizzo:.0%} utilization: {capacita / 1e6:5.1f} million tokens/month")
print()
print("API price (EUR/Mtoken)  break-even (Mtoken/month)  verdict")
for prezzo in (0.1, 1.0, 10.0):
    pareggio = costo_server_mese / prezzo        # million tokens per month
    if pareggio * 1e6 <= capacita:
        verdetto = "the server pays for itself at the assumed utilization"
    elif pareggio * 1e6 <= capacita_100:
        verdetto = "it pays for itself only by pushing utilization"
    else:
        verdetto = "it does not get there even at full load"
    print(f"{prezzo:>22.1f}  {pareggio:>25.1f}  {verdetto}")

# capacity at full load:  25.9 million tokens/month
# capacity at 20% utilization:   5.2 million tokens/month
#
# API price (EUR/Mtoken)  break-even (Mtoken/month)  verdict
#                    0.1                      400.0  it does not get there even at full load
#                    1.0                       40.0  it does not get there even at full load
#                   10.0                        4.0  the server pays for itself at the assumed utilization

With these assumptions the server produces at most 25.9 million tokens per month, 5.2 at 20% utilization. At 0.1 and 1 euro per million tokens the break-even (400 and 40 million) is beyond the capacity: the metered API costs less, and on-prem is chosen for data sovereignty and not for price. Only with a price of 10 euros per million does the break-even (4 million) fall within the assumed utilization. The conclusion changes with your numbers, which is why the calculation must be redone: faster hardware moves the break-even, a cheaper model pushes it away. CPU throughput must be measured on your own load, because it depends on the model, its quantization and the prompt length.

Trade-off, stated plainly: what you give up choosing on-prem

Honesty about the trade-off, not marketing: choosing on-prem is not free in an absolute sense, it is free on the monthly invoice. Three real costs remain. First, the model ceiling: a local 8B-parameter model does not compete with frontier models on complex reasoning or advanced multimodal tasks, and there is no way around it without changing hardware. Second, the operational burden: security patches, model updates, monitoring, backups, everything a cloud provider handles for you stays on your plate when the infrastructure is yours. Third, network latency depends on local hardware capacity rather than the scale of a dedicated data center: on a single VPS shared across multiple services, a load spike has to be absorbed by the same machine that also serves the application backend. In exchange you get full data sovereignty and a fixed, predictable invoice that does not depend on request volume.

When cloud is still the right choice

On-prem is not a dogma, it is a default for the average case. If the workload needs multi-step reasoning over very long documents, advanced multimodal generation, or burst scale that a single VPS cannot absorb, cloud remains the technically correct choice, as long as you accept its variable cost and the transfer of data to a third party. The useful question for a decision-maker is not "on-prem or cloud" in the abstract, but "does this specific workload actually need the capacity ceiling only cloud offers, or am I paying for that ceiling and never using it".

Data sovereignty: why it outweighs price in certain sectors

For companies handling healthcare, legal, or financial data with a legal basis in Europe, the question "where does my data go when I call the API" is not secondary: it is a compliance requirement under GDPR, with direct implications on international transfers and controller liability. With on-prem AI, the answer is structurally simple: the data never leaves the machine that hosts it, because there is no network call to a third-party processor. It is not a marketing advantage, it is a change in risk surface: fewer third parties in the processing chain, fewer contracts to verify, fewer questions to answer in an audit.

Two articles of Regulation (EU) 2016/679 give the measure. Article 28 says the controller uses only processors providing sufficient guarantees to implement appropriate technical and organisational measures; Article 44 opens the chapter on transfers of personal data to third countries with a general principle to which every transfer is subject. Which rule applies to a concrete case, and whether a given configuration of the data satisfies it, is decided by your legal counsel or the data protection officer: this article is not legal advice.

What happened next: the case that tests the thesis

On 23 October 2026 this site's ml-service moved from on-prem Ollama to OpenRouter, free plan: primary model recursive/flint-v1.0:free with fallback openrouter/free. It stayed there until 12 November 2026: since the deployment of that day the site uses a local AI engine on its own server, in Europe, with a job queue, and AI requests are not sent to third-party providers. On a CPU with no GPU the times are longer (measured generation: 9-13 tokens per second). The assessment that follows concerns the OpenRouter period. The monthly cost of AI remained zero, so the comparison is not played out on the invoice. The repo does not document the detailed reason for the choice; it does document what changed, and that is enough to redo the trade-off calculation with real numbers.

What you gain

• Latency. In the backend comments, generation by llama3.1:8b on CPU was estimated at 10-60 seconds per answer, so much that it required a dedicated 75-second timeout for chat alone. The typical call on the new provider was measured on 23 October at 1.5-2 seconds. It is the "capacity ceiling" discussed in the trade-off section, seen up close.
• Free CPU. Inference no longer competes with Spring Boot, PostgreSQL and Redis on the same machine: the risk of the "spike to be handled with the same machine" disappears.

The two items describe the same shift seen from two sides: moving inference off a shared machine with no accelerator shortens the wait for whoever asks and frees resources for the services that remain. As a general criterion, the time of the slow requests matters more than the average one, because they are the only ones the user notices and for which they choose to leave.

What you pay, plainly

• Data leaves. Every request carried data_collection: "deny" in the request body, in the code and not in the account panel, so the constraint is verifiable by a test. But Zero Data Retention is not obtainable on the free plan (measured: 404 error "No endpoints found matching your data policy") and openrouter/free has no in-region EU routing. The honest formula is therefore "prompts are not used for training", never "they are not retained".
• Rate limits. The free plan answered 429 under load. The ml-service retried with backoff up to 4 attempts of 60 seconds each, so the worst case far exceeded the 75-second timeout on the backend side. When the provider did not answer, the tools returned the deterministic result with an "AI unavailable" notice, without automatically falling back to another cloud.
• A reasoning model behaves differently. With a low token budget it emits no content: for this reason the provider had a minimum threshold of 256 tokens and used the reasoning text when the content was empty, recording it in the logs. With a local model of your choosing this class of surprises does not exist.
• Dependence on a catalog you do not control. A :free identifier can change or disappear. With Ollama the model was a file on disk.

The four items have a common root: moving inference to a third party transfers control as well as effort. Rate limits and dependence on a catalog you do not choose are availability risks, and are covered with a declared fallback; the data question is a legal risk, to be assessed with Articles 28 and 44 of the regulation in hand (see the section on sovereignty) and with a professional.

What it means for the decision-maker

The article's criterion holds: the question remains "does this workload need the capacity ceiling or data sovereignty?". But the answer depends on the class of data that enters the prompt, not on the principle. In the OpenRouter period, for a public chat and for tools that work on text entered by the user, the loss of perimeter weighed little and latency weighed a lot, and the balance leaned toward the cloud. For the healthcare, legal or financial data of the sovereignty section the balance remains the original one. One principle remains standing, whichever side is chosen, and it is non-negotiable: the site declares the provider it actually uses. The mistake to avoid is not choosing the cloud, it is going on writing "on-prem" after switching it off.

Who this approach is for

The practical advice for a decision-maker: if the company's AI workload is mostly made of non-capability-critical tasks (internal search, classification, summarization, structured extraction), and the company operates under strict data residency or compliance obligations, on-prem AI on a single VPS is not a downgrade compromise: it is the more defensible choice on predictable cost and risk surface. Reserve cloud for the cases that genuinely need the higher capacity ceiling, not as the default for every AI call the company makes.

What you can do now

You can redo the calculation between your own server and a metered API with your own numbers, and you can tell a price choice from a data sovereignty choice. You know which GDPR articles to consult before sending personal data to a third party, and that the decision belongs to a professional. The previous article, Three Production Postmortems, showed what happens when an implicit rule is violated.

The series in fourteen articles

Fourteen articles, in an order that matters, because each one uses the concepts of the previous:

1. 01 - Introduction to Platform Engineering: What is an IDP and Why It Matters
2. 02 - Architecture of a Modern IDP: Components and Integration
3. 03 - Golden Paths: Definition and Implementation of Standardized Flows
4. 04 - Backstage.io: Platform Portal for Developer Self-Service and Service Catalog
5. 05 - Infrastructure as Code Patterns: Terraform, Pulumi, and Policy as Code
6. 06 - Service Catalog and CMDB: Service Management and Governance
7. 07 - Platform Observability: DORA Metrics and SPACE Framework
8. 08 - Multi-Cloud Platform Engineering: Portability and Vendor Lock-in
9. 09 - Developer Experience and Feedback Loops: Qualitative and Quantitative Metrics
10. 10 - AI Integration in Platforms: Intelligent Deployments and Self-Healing Infrastructure
11. 11 - Platform Security and Compliance: Zero Trust and Policy Enforcement
12. 12 - Case Study: End-to-End IDP Implementation for Tech Startup
13. 13 - Three Production Postmortems: Flyway, JAR Deploy, and Duplicate Beans
14. 14 - AI On-Prem vs Cloud: Privacy, Cost, and Data Sovereignty

The sequence has three blocks. The first seven (from the foundations to measurement) build the platform: why it is needed, how it is composed, the guided paths, the portal, the infrastructure code, the catalog and the metrics. Articles eight to twelve bring it to maturity: portability across providers, developer experience, automation with a brake, security and an implementation plan. The last two look at production and architecture decisions. A possible short path is the first, the ninth and the twelfth, which say why a platform is needed, how to listen to developers and how to plan the work.