The problem: energy costs of LLM inference

A model like LLaMA 3.1 (8B parameters) requires ~800 MiB of memory and ~100 billion operations to generate 1000 tokens. On cloud GPU (AWS p3.2xlarge), this means ~$0.04 per request at public AWS pricing. But the real energy cost is higher: 40-60 kWh per million tokens equal ~0.8 kg CO₂ per request (with average European grid), plus data center cooling overhead (+40%).

In enterprise and personal contexts, the solution is not "use less AI": it's achieving the same result with less energy. Four levers:

• Quantization: reduce bit-width (float32 → int8 → int4) without measurable accuracy loss
• Local inference: Ollama on-prem avoids cloud roundtrip, latency, and cloud energy envelope
• KV cache compression: optimize memory activation during generation
• Pruning and distillation: reduce parameters/depth while maintaining performance

Technique 1: Quantization (INT8, INT4, GPTQ)

Quantization is the first step. A float32 weighs 4 bytes; int8 weighs 1 byte. This reduces memory 4x and operations accordingly. The trick is maintaining accuracy: neural networks are robust to discretization if done correctly.

INT8 Symmetric Quantization (simple, used by Ollama)

import numpy as np

def quantize_int8(weights):
    """Convert float32 weights to int8 with symmetric quantization."""
    scale = np.max(np.abs(weights)) / 127.0
    quantized = np.clip(np.round(weights / scale), -128, 127).astype(np.int8)
    return quantized, scale

def dequantize_int8(quantized, scale):
    """Convert int8 back to float32."""
    return quantized.astype(np.float32) * scale

# Example
weights = np.random.randn(1000, 1000).astype(np.float32)
q_weights, scale = quantize_int8(weights)
reconstructed = dequantize_int8(q_weights, scale)
print(f"Memory reduction: {weights.nbytes / q_weights.nbytes:.1f}x")
print(f"MSE reconstruction: {np.mean((weights - reconstructed)**2):.6f}")

INT4 with GPTQ (production-ready)

GPTQ (Gradient Quantization after Training) quantizes to 4 bits (16 levels per weight) using Hessian to find optimal scaling. It's the 2026-2027 standard for efficient serving. AutoGPTQ library makes it trivial:

from auto_gptq import AutoGPTQForCausalLM
from transformers import AutoTokenizer

# Load pre-quantized GPTQ model
model_name = "TheBloke/Llama-2-7B-GPTQ"  # public alias
model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device="cuda:0",
    use_triton=False,  # Ollama doesn't use Triton, but you can on native GPU
    inject_fused_attention=False
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Inference identical to float32, but 4x less memory
inputs = tokenizer("What is energy efficiency?", return_tensors="pt")
output = model.generate(**inputs, max_length=50)
print(tokenizer.decode(output[0]))

Numerical result: LLaMA 3.1 8B goes from 16GB (float16) to 4GB (int4 GPTQ). Inference latency increases ~5-10% (depends on kernel); effective throughput often improves because batch size can increase. Energy: linear reduction with memory, ca. 3-4x less.

Technique 2: KV cache optimization

During generation, the model attends to KV (key-value) cache of the previous sequence to compute attention of the next token. For a 4K token sequence, KV cache occupies ~2-3 GB (depends on batch size). Compression techniques:

Sparse KV Cache with FlashAttention-2

Alternate strategy: store only K,V for "important" tokens (e.g., every 4th token for remote context). Works well on code/technical documents where some tokens hold more information:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Enable FlashAttention-2 + sparse KV cache
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b",
    torch_dtype=torch.float16,
    attn_implementation="flash_attention_2"  # requires GPU + flash-attn install
)

# Inference with reduced KV memory
with torch.inference_mode():
    inputs = tokenizer("Explain pruning", return_tensors="pt").to("cuda")
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        # KV cache automatically sparse with FA2
    )

Quantized KV Cache (INT8)

Quantize K,V to int8 on-the-fly. Reduces cache memory 4x. Trade-off: loses ~0.5-1% accuracy on long-context tasks (2K+ tokens):

def quantize_kv_cache(k_cache, v_cache):
    """Quantize KV cache to int8 to reduce memory."""
    def _quantize(x):
        scale = torch.max(torch.abs(x)) / 127.0
        return torch.clip(torch.round(x / scale), -128, 127).to(torch.int8), scale

    k_q, k_scale = _quantize(k_cache)
    v_q, v_scale = _quantize(v_cache)
    return k_q, v_q, k_scale, v_scale

# Usage in inference loop
for step in range(max_steps):
    k, v = model.forward_kv(input_ids)  # pseudo
    k_q, v_q, k_s, v_s = quantize_kv_cache(k, v)
    # Use k_q, v_q in next step + scale factors

Technique 3: Model pruning and distillation

Reduce model parameters. A LLaMA 3.1 7B has 7 billion parameters; through structured pruning, you can reduce ~30-50% without measurable accuracy loss.

Magnitude Pruning

Remove weights close to zero. Simple but effective:

import torch

def magnitude_prune(model, sparsity=0.3):
    """Prune 30% of weights with smallest magnitude."""
    for name, param in model.named_parameters():
        if 'weight' in name and len(param.shape) >= 2:
            # Calculate threshold
            sorted_weights = torch.sort(torch.abs(param.flatten()))[0]
            threshold = sorted_weights[int(len(sorted_weights) * sparsity)]

            # Apply mask
            mask = torch.abs(param) > threshold
            param.data *= mask
    return model

# After pruning, fine-tune on real dataset to recover ~2-3% accuracy
pruned_model = magnitude_prune(model, sparsity=0.3)
# Fine-tuning loop...

Knowledge Distillation (more sophisticated)

Train a small model (e.g., 1.5B) to imitate a large one (7B). The small one uses ~75% less energy:

from torch.nn import functional as F

def distillation_loss(student_logits, teacher_logits, labels, temperature=4.0, alpha=0.7):
    """Loss combining classification + distillation."""
    ce_loss = F.cross_entropy(student_logits, labels)

    # KL divergence between soft distributions (temperature scaling)
    student_probs = F.log_softmax(student_logits / temperature, dim=-1)
    teacher_probs = F.softmax(teacher_logits / temperature, dim=-1)
    kl_loss = F.kl_div(student_probs, teacher_probs, reduction='batchmean')

    return alpha * ce_loss + (1 - alpha) * kl_loss * (temperature ** 2)

# Training loop
for batch in train_loader:
    inputs, labels = batch
    student_out = student_model(inputs)
    teacher_out = teacher_model(inputs).detach()

    loss = distillation_loss(student_out, teacher_out, labels)
    loss.backward()
    optimizer.step()

Local inference with Ollama: energy-optimal setup

Ollama is an on-prem runtime that loads quantized models and serves inference. Eliminates cloud roundtrip: <100ms latency vs 500ms+ cloud, and pure energy on your hardware (no cloud tax).

Basic Ollama setup

# Install Ollama (macOS, Linux, Windows + WSL2)
curl https://ollama.ai/install.sh | sh

# Download quantized model (e.g., Llama 2 7B int4, ~4GB)
ollama pull llama2:7b  # or mistral, neural-chat, etc.

# Start server (defaults to port 11434)
ollama serve

Python client for local inference

import requests
import json

def infer_ollama(prompt, model="llama2:7b", temperature=0.7):
    """Run inference on local Ollama."""
    url = "http://localhost:11434/api/generate"
    payload = {
        "model": model,
        "prompt": prompt,
        "temperature": temperature,
        "stream": False
    }

    response = requests.post(url, json=payload)
    result = response.json()
    return result["response"]

# Example
output = infer_ollama("Briefly explain machine learning")
print(output)

# Benchmark energy: ~0.5 kWh per 100K tokens (on modern CPU, no GPU)
# vs ~10-15 kWh via cloud API = 20-30x less energy

Streaming for real-time inference

import requests
import json

def infer_ollama_stream(prompt, model="llama2:7b"):
    """Stream output token by token."""
    url = "http://localhost:11434/api/generate"
    payload = {
        "model": model,
        "prompt": prompt,
        "stream": True
    }

    with requests.post(url, json=payload, stream=True) as r:
        for line in r.iter_lines():
            if line:
                chunk = json.loads(line)
                print(chunk["response"], end="", flush=True)

# Useful for interactive applications (chatbots, APIs)

Benchmarking: key energy metrics

It's not enough to measure latency; you must actually measure energy.

Efficiency metrics

• Tokens/Joule: how many tokens you generate per unit energy. 2027 target: >1000 tok/J for on-prem
• Throughput: tokens/second (latency isn't everything; batch size counts)
• Memory peak: GB of RAM/VRAM used (limits scalability)
• Carbon intensity: gCO₂ per 1000 tokens (depends on energy grid)

Benchmark script

import time
import psutil
import requests

def benchmark_ollama(prompt, model="llama2:7b", num_trials=5):
    """Measure throughput, latency, memory."""
    times = []
    memory_peaks = []

    for _ in range(num_trials):
        process = psutil.Process()
        mem_before = process.memory_info().rss / 1e9  # GB

        start = time.time()
        response = requests.post(
            "http://localhost:11434/api/generate",
            json={"model": model, "prompt": prompt, "stream": False}
        )
        elapsed = time.time() - start

        mem_after = process.memory_info().rss / 1e9

        times.append(elapsed)
        memory_peaks.append(mem_after - mem_before)

    avg_time = sum(times) / len(times)
    avg_mem = sum(memory_peaks) / len(memory_peaks)
    tokens = len(response.json()["response"].split())  # rough estimate

    print(f"Avg latency: {avg_time:.2f}s | Tokens: {tokens} | Throughput: {tokens/avg_time:.1f} tok/s")
    print(f"Memory peak: {avg_mem:.2f} GB")
    print(f"Efficiency: {tokens/avg_time/avg_mem:.1f} tok/s/GB")

benchmark_ollama("Explain deep learning")

Orchestration: batch inference on CPU

Ollama by default leverages GPU if available; but on CPU, parallelization is limited. For batch workflows (e.g., indexing, report generation), orchestrate with multiprocessing:

from concurrent.futures import ProcessPoolExecutor
import requests

def process_document(doc_id, text):
    """Summarize a document via Ollama."""
    payload = {
        "model": "mistral",
        "prompt": f"Summarize in 2 sentences:\n{text}",
        "stream": False
    }
    response = requests.post("http://localhost:11434/api/generate", json=payload)
    return {"doc_id": doc_id, "summary": response.json()["response"]}

# Batch of documents
docs = [
    {"id": 1, "text": "..."},
    {"id": 2, "text": "..."},
    # ...
]

with ProcessPoolExecutor(max_workers=2) as executor:  # Adjust based on cores
    results = list(executor.map(
        lambda d: process_document(d["id"], d["text"]),
        docs
    ))

print(results)

Trade-off: accuracy vs energy

No technique is a "free lunch". Summary of 2027 trade-offs:

2027 strategy: start with INT4 GPTQ (quick, immediate ROI) + Ollama on-prem. If accuracy drops measurably, add pruning fine-tuning. Distillation is for mission-critical edge deployment.

Case study: implementation on Federico's portfolio

This site uses Ollama on-prem on Hetzner VPS CCX13 (4 vCPU, 8GB RAM). Loads summarization model (Mistral 7B int4) for AI recommendations feature on articles:

Architecture

• Frontend Angular calls backend REST endpoint
• Backend Spring Boot (Java 21) makes request to Ollama via HTTP (localhost:11434)
• Ollama responds in <1 sec (on-prem latency)
• No cloud cost, no external roundtrip

Configuration (systemd unit)

[Unit]
Description=Ollama LLM Inference Server
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/ollama serve
Restart=always
User=ollama
Environment="OLLAMA_MODELS=/opt/ollama/models"
Environment="OLLAMA_KEEP_ALIVE=5m"

[Install]
WantedBy=multi-user.target

Energy consumption: ~15-20W idle, ~60W during batch inference. Annual cost: <€30 (vs ~€8000 on equivalent cloud).

Common problems and solutions

Problem 1: OOM (Out of Memory) with 7B+ models on CPU

Solution: use INT4 quantization (mandatory); reduce batch size to 1; enable mmap in Ollama. If still insufficient, use distilled model (1.5B).

Problem 2: Accuracy drops too much with INT4

Solution: verify quantized model is GPTQ-trained (not just converted). Models from TheBloke repo (HuggingFace) are reliable. Otherwise, quantize yourself on specific dataset (AutoGPTQ library).

Problem 3: Ollama inference latency is 10x worse than cloud

Solution: waiting is OK if batch. For online inference, add caching (Redis) and pre-compute embeddings. Or migrate to local GPU (RTX 4070 ~€500, 200x faster than CPU).

Problem 4: Token distribution shift

Situation: quantized model perfect on benchmarks, but output degrades on real texts. Solution: calibrate quantization scale on in-domain dataset (not just benchmark). Use calibration set of 100-500 real samples.

Energy roadmap 2027-2028

Technology predictions:

• Speculative Decoding (early 2027): produces multiple tokens per forward pass using mini-model drafting. 2-3x latency reduction without accuracy loss
• Mixture of Experts (MoE) inference: activate only subset of parameters. Models like Mixtral become standard; 5x energy saving
• Hardware quantization-native (2027+): new chips (e.g., SambaNova, Cerebras) with native INT4 compute. 10-100x more efficient than float32 GPU
• On-device LLM (edge): distilled <1B models on smartphone. Battery for 10K tokens, zero latency, zero privacy risk

FAQ

1. Does quantization cause non-deterministic output?

No. Quantization is deterministic (same input = same output). But small numerical variations can amplify over long sequences. If absolute reproducibility is needed, use seed fixing + float32 computation path in parallel (benchmarking only).

2. Can I quantize a custom fine-tuned model?

Yes, but it requires calibration. Use AutoGPTQ with custom dataset (~100-500 samples from your distribution). Result: INT4 with <1% accuracy loss.

3. Is Ollama production-ready?

For read-only inference, yes. For systems that write (e.g., RAG with real-time embedding updates), it's stable but young. Monitoring with rate limiting + circuit breaker (backend) recommended.

4. Is on-prem always better, or are there cloud cases?

Cloud is better if: (a) unpredictable traffic bursts (elastic scaling), (b) heterogeneous models (don't want to manage 20 different containers), (c) compliance no-data-residency. Otherwise on-prem wins on cost + latency + privacy.

5. How do I monitor real energy during inference?

On Linux: `turbostat`, `powertop` (kernel profiling). On specific HW: smart PDU (smartctl). Integrated metric: Python `pyjoule` library tracks CPU time + memory bandwidth → joule estimate. Cloud benchmark: use their cost model (AWS Pricing API).

Conclusion

In 2027, optimizing AI energy is not an ethical choice—it's an economic one. INT4 quantization, KV cache compression, and Ollama on-prem inference reduce footprint 10-30x vs conventional cloud, with low technical investment. Implement step by step: GPTQ quantization → on-prem Ollama → pruning fine-tuning if accuracy drops. The code in this guide scales from prototype to production with minimal changes.

Next article in series: "Sustainable AI"—Carbon accounting frameworks and green LLM training 2027.