05 - LLM APIs in Production: OpenAI, Anthropic, and Open Source Models

Why the call that works in the playground is not enough in production

A call to a language model, in a test notebook, almost always succeeds. In production the same call fails for reasons that have nothing to do with your code: the provider is overloaded, you have exceeded your quota, the network times out halfway through a response. The real work is not writing the call, it is deciding what your service does when the call goes wrong, and what each choice costs you, in money and in time.

This article takes four decisions one at a time: when to retry and when not to, when to reuse a response you already have, when to switch to another provider, how to keep spending under control. For each one you get a mechanism in a few lines of Python. The listings run against a fake provider that fails according to a script and a simulated clock: no network, no key, no model called. They are a teaching model of the control logic, not a production-ready client: the exception names of the real SDKs change from one provider to the next, and those have to be read in their own documentation.

If you have not yet decided whether you need a model of your own or an API, start from Fine-Tuning LLMs: LoRA, QLoRA and PEFT, which explains when customizing a model is justified. If instead the problem is the quality of the answers and not their reliability, you first need Advanced Prompt Engineering.

Telling errors apart: which ones to retry and which not

The first mistake is to treat every failure the same way. HTTP errors fall into two families with opposite remedies. The transient ones go away by themselves if you wait: the provider is momentarily overloaded or you have exceeded the allowed rate. The permanent ones do not go away: the request is malformed, the key is wrong, the credit has run out. Retrying a permanent error does not fix it, and it makes you pay, in time and sometimes in money, for a certainty you already had.

The two official documentation sets I read agree on this point. Anthropic's errors page lists as retryable the 429 (rate limit), the 500 and the 529, which signals a general overload of the API. OpenAI's lists the 429 for requests that are too fast, the 500 and the 503 for an overloaded model, with the instruction to honor the Retry-After header when it is present. Both, however, contain a treacherous case.

• 429 for rate: you are sending requests too fast, wait and retry.
• 429 for spending cap or exhausted credit: Anthropic writes that the 429 from a spending cap has no retry-after and keeps failing until access resumes; OpenAI has a separate 429 for exhausted credit.
• 500, 503, 529: a problem on the provider's side, often brief, retried with a growing wait.
• 400, 401, 403, 413: the request or the credentials are wrong, retrying changes nothing.

The same number, 429, can therefore mean "wait two seconds" or "it will not work again until you top up the account". A retry loop that looks only at the number burns its attempts on the second case and delays the alarm that should wake someone up. The practical rule is to read the error code in the body of the response as well, not only the HTTP status.

Retry with growing waits and random variation

The wait between one attempt and the next must not be fixed. If a thousand clients fail at the same instant and all wait one second, they come back together and make the provider fail again. Exponential backoff doubles the wait at each attempt; jitter makes it random, so the clients spread out over time. An article on the AWS Architecture Blog compares the variants: "Full Jitter", in which the wait is a random number between zero and the growing ceiling, does less total work than "Equal Jitter" and turns out faster, while the decorrelated variant makes slightly more calls and takes slightly less time, with no clear winner. Here I use Full Jitter, the simplest to write.

The listing has two parts. The first classifies errors: is_retryable says no to the 429 with code spend_cap, and yes to the transient statuses. The second is the loop: at each failure it computes the wait with rng.uniform(0, ...) and takes the larger of that and Retry-After, because if the provider tells you to wait two seconds, a random wait of half a second would be a discourtesy that costs you another 429 response. RFC 6585 defines the 429 and says the response "MAY" include Retry-After: it is optional, so your code must also work when it is missing.

import random

class ApiError(Exception):
    """HTTP error as a provider's client would see it."""
    def __init__(self, status, code, retry_after=None):
        super().__init__(f"{status} {code}")
        self.status, self.code, self.retry_after = status, code, retry_after

def is_retryable(err):
    if err.status == 429 and err.code == "spend_cap":
        return False                   # a spending cap does not pass by waiting
    return err.status in (408, 429, 500, 502, 503, 529)

def call_with_retry(fn, clock, rng, max_attempts=4, base=1.0, cap=20.0):
    for attempt in range(max_attempts):
        try:
            return fn()
        except ApiError as err:
            if not is_retryable(err) or attempt == max_attempts - 1:
                raise
            full_jitter = rng.uniform(0, min(cap, base * 2 ** attempt))
            wait = max(full_jitter, err.retry_after or 0)   # Retry-After wins
            print(f"[retry] attempt {attempt + 1}: {err} -> wait {wait:.2f}s")
            clock["t"] += wait         # fake clock: no real sleep

class FakeProvider:
    """Fails according to a script, then answers. No network, no model."""
    def __init__(self, script):
        self.script, self.calls = list(script), 0
    def __call__(self):
        self.calls += 1
        if self.script:
            raise self.script.pop(0)
        return "response"

The fake provider is the tool that makes the behavior verifiable without spending a euro: it receives a script of errors and raises them one at a time. The waits are simulated with a clock that simply advances by a number, so the listing finishes instantly. The scenarios block at the end of the section on costs runs all of it, and its outputs are the ones I comment on from here on.

In the first scenario the provider fails twice with an error and then succeeds: the client waits 2.00 seconds, which is the Retry-After exceeding the random number, then 0.30, and on the third call it gets the response. In the second, the 429 from a spending cap stops everything at once: a single call, no attempt wasted. In the third, the 503 persists and the loop stops at the fourth attempt, with waits of 0.65, 0.14 and 2.14 seconds. The waits change with the seed of the random generator: they are figures from this run, not values to expect in production.

One trap worth naming: the official clients already retry on their own. Anthropic's documentation says the SDK retries transient errors with exponential backoff, twice by default, and accepts max_retries to change or switch it off. If you put a loop of your own with four attempts on top of it without zeroing that value, each of your attempts can become three HTTP calls: in the worst case twelve requests for a single prompt, which on a rate-limit 429 make the problem worse. Choose a single level that retries and set the other to zero. Check the default value of your SDK in its documentation, and do not assume it is the same as another one's.

Cache: reusing a response only when it is legitimate

A cache avoids paying twice for the same response. The idea is simple: you compute a fingerprint of the request and, if that fingerprint has been seen before, you return the saved response. The delicate point is what goes into the fingerprint. The model, the messages and the temperature must all be in it: if you change even a single comma in the system prompt, the old response is no longer valid.

Saving responses generated at temperature 0.7 as well is a design mistake: at positive temperature the model samples, and two identical calls can give different texts, so the cache freezes just one of the possible responses and presents it as "the" response. The listing skips the cache when temperature is greater than zero. Even at temperature zero, do not take for granted that two identical calls produce identical texts: check it in the provider's documentation. A cache should be designed for questions that admit a stable answer: classifications, extractions, summaries of a fixed document.

import hashlib, json

class LLMCache:
    def __init__(self):
        self.store, self.hits, self.misses = {}, 0, 0

    def key(self, model, messages, temperature):
        raw = json.dumps([model, messages, temperature], sort_keys=True)
        return hashlib.sha256(raw.encode()).hexdigest()

    def ask(self, model, messages, temperature, call):
        if temperature > 0:            # variable output: it is not reused
            return call()
        k = self.key(model, messages, temperature)
        if k in self.store:
            self.hits += 1
            return self.store[k]
        self.misses += 1
        self.store[k] = call()
        return self.store[k]

json.dumps(..., sort_keys=True) makes the fingerprint independent of the order of the fields, and sha256 reduces it to a fixed-length string without revealing the prompt in the key. In the test in the scenarios block, three identical requests at temperature zero produce a single real call (one miss and two hits), while the two at temperature 0.7 both go through the model: three real calls out of five in total. This is a count on my example, not an estimate of the saving you will get: the percentage depends on how many of your service's questions really repeat, and the only way to know it is to count the hits in operation.

Two limits need stating. The in-memory cache disappears at every restart and is not shared between instances: for a service with several processes you need an external store and an expiry (TTL) for each entry. And the user's personal data must not end up in the fingerprint unless you have decided how long to keep it: a saved response is stored data.

Fallback between providers: useful, but with a condition

If the main provider is down, a second provider keeps the service up. The cost is real: you have to maintain two integrations, two message formats, and accept that the second one's responses will not have the quality of the first's. A fallback pays off when unavailability costs you more than the difference in quality, for instance in an internal assistant where a mediocre answer is better than no answer.

The condition is in the listing: route moves on to the next provider only if the error depends on the provider (overload, unavailability). If the status is 400, 401, 403 or 413, the fault lies with the request or the key and the second provider would fail in the same way: the error is re-raised. A router that catches Exception and tries the next one hides the defects of your own code behind a fallback that "works" with a different model from the one you thought you were using.

def route(providers, messages):
    errors = []
    for name, fn in providers:
        try:
            return name, fn(messages)
        except ApiError as err:
            if err.status in (400, 401, 403, 413):
                raise                  # the request's fault: another provider will not help
            errors.append(f"{name}: {err}")
    raise RuntimeError("all providers failed: " + "; ".join(errors))

The router also returns the name of the provider that answered: ('secondary', 'ok from the second'). Recording that name in every log is essential, otherwise you will not be able to say why quality dropped on Tuesday afternoon. In the second scenario of the block, a 400 passes through the router without trying the second provider. A limit remains that the code does not cover: if the first provider is slow rather than down, the router waits for the timeout before moving on, and every user pays for that wait. A circuit breaker that skips the provider after a series of failures solves the problem, but it is one more piece to maintain.

Keeping spending under control

Providers bill by the token, separating input tokens from output tokens, and prices change over time: for this reason the article reports no price list. A price list written here would be out of date at the first revision. The listing's tracker receives the prices as a parameter and uses invented numbers (0.50 and 2.00 dollars per million tokens for the "small" model, 5.00 and 20.00 for the "large" one): they serve to show the calculation, not to estimate any invoice. The real prices are on the provider's page, and must be re-read before every budget forecast.

class UsageTracker:
    def __init__(self, prices, daily_budget):
        self.prices, self.budget, self.spent = prices, daily_budget, 0.0

    def log(self, model, tokens_in, tokens_out):
        p_in, p_out = self.prices[model]          # price per million tokens
        cost = (tokens_in * p_in + tokens_out * p_out) / 1_000_000
        self.spent += cost
        if self.spent > self.budget * 0.8:
            print(f"[cost] alarm: {self.spent:.4f} of {self.budget} USD")
        return cost

The formula is (tokens_in * price_in + tokens_out * price_out) / 1_000_000. With the fake prices, the first call (2000 input tokens, 500 output tokens) costs 0.0020 dollars and the second, with the large model, 0.0310: the difference between the two models, for the same length, is an order of magnitude, and it is the reason why choosing the smallest model able to do the job is the strongest cost lever of all. With a budget of 0.05 the eighty percent threshold (0.04) trips on the third call, which brings the total to 0.0530.

The number of tokens is read from the provider's response, which returns it in the usage object: counting it by hand on the text would give a different value, because each model family has its own tokenizer. An alarm that trips after the spending is a warning, not a brake: for a real limit you need a check before the call, which rejects the request when the budget is exhausted, and a spending cap set in the provider's dashboard, which is the only protection that works even when your code has a defect.

The scenarios, run one after the other

The last block puts the four pieces to the test with the fake provider. Each line of the output carries in square brackets the section it belongs to, so you can connect it to the mechanism you have just read: [retry] for the three error scenarios, [cache] for the count of real calls, [router] for the fallback and [cost] for the spending. The fixed seed of the random generator makes the run repeatable, and the printed waits are those of that seed.

if __name__ == "__main__":
    rng, clock = random.Random(7), {"t": 0.0}

    p = FakeProvider([ApiError(429, "rate_limit", retry_after=2), ApiError(503, "overloaded")])
    print("[retry] outcome:", call_with_retry(p, clock, rng), "after", p.calls, "calls")

    p = FakeProvider([ApiError(429, "spend_cap")] * 5)
    try:
        call_with_retry(p, clock, rng)
    except ApiError as err:
        print(f"[retry] {err}: no new attempt, calls = {p.calls}")

    p = FakeProvider([ApiError(503, "overloaded")] * 9)
    try:
        call_with_retry(p, clock, rng)
    except ApiError as err:
        print(f"[retry] {err}: attempts exhausted, calls = {p.calls}")

    cache, n = LLMCache(), {"calls": 0}
    def llm():
        n["calls"] += 1
        return f"response #{n['calls']}"
    m = [{"role": "user", "content": "What is an index?"}]
    for _ in range(3):
        cache.ask("model-x", m, 0, llm)
    cache.ask("model-x", m, 0.7, llm)
    cache.ask("model-x", m, 0.7, llm)
    print(f"[cache] real calls = {n['calls']}, hits = {cache.hits}, misses = {cache.misses}")

    def primary(msgs): raise ApiError(529, "overloaded")
    def secondary(msgs): return "ok from the second"
    print("[router]", route([("primary", primary), ("secondary", secondary)], m))
    def bad(msgs): raise ApiError(400, "invalid_request")
    try:
        route([("primary", bad), ("secondary", secondary)], m)
    except ApiError as err:
        print(f"[router] {err}: propagated without switching provider")

    t = UsageTracker({"small": (0.50, 2.00), "large": (5.00, 20.00)}, daily_budget=0.05)
    for model, i, o in [("small", 2000, 500), ("large", 3000, 800), ("large", 2000, 500)]:
        print(f"[cost] {model}: {t.log(model, i, o):.4f} USD")
    print(f"[cost] total {t.spent:.4f} USD")

# Output:
# [retry] attempt 1: 429 rate_limit -> wait 2.00s
# [retry] attempt 2: 503 overloaded -> wait 0.30s
# [retry] outcome: response after 3 calls
# [retry] 429 spend_cap: no new attempt, calls = 1
# [retry] attempt 1: 503 overloaded -> wait 0.65s
# [retry] attempt 2: 503 overloaded -> wait 0.14s
# [retry] attempt 3: 503 overloaded -> wait 2.14s
# [retry] 503 overloaded: attempts exhausted, calls = 4
# [cache] real calls = 3, hits = 2, misses = 1
# [router] ('secondary', 'ok from the second')
# [router] 400 invalid_request: propagated without switching provider
# [cost] small: 0.0020 USD
# [cost] large: 0.0310 USD
# [cost] alarm: 0.0530 of 0.05 USD
# [cost] large: 0.0200 USD
# [cost] total 0.0530 USD

If you remove the is_retryable check from the second scenario, the calls rise from one to four without gaining anything: it is the quickest way to see the cost of a blind retry. If instead you let route catch every exception, the line with 400 invalid_request becomes an error on all providers and you lose the information about who got it wrong.

Streaming and models of your own: what changes

Streaming returns the text as the model generates it, and lowers the time the user perceives: they see the first words appear instead of staring at a waiting indicator. The price is in error handling. Anthropic's documentation warns that with streaming over server-sent events an error can arrive after a 200 response: your retry loop, which looks at the HTTP status, does not see it. You have to handle the error event inside the stream and decide what to do with the text already shown, typically leaving it and flagging that the response is incomplete.

Running an open model on your own hardware shifts the costs: you do not pay per token, but you pay for machines, energy and the time of whoever runs them, and the capacity is the one you have bought. The main advantage is control over the data, which stays in your environment; the drawback is that quality and speed depend on the model you manage to run. The choice is not binary: a mixed architecture puts the most capable model on the hard requests and a small model, your own or paid, on the repetitive ones. Which ones are worth trying is decided with a test on your own data, not with a ranking.

What you can do now

You can separate the errors that get retried from those that do not, and you know why the 429 is not enough to decide. You can write a retry with growing waits and random variation that honors Retry-After, and you can avoid multiplying attempts by stacking your loop on top of the SDK's. You know when a cache is legitimate and when it freezes a random response. You know that a fallback on provider errors alone protects the service without hiding your own defects, and you can keep track of the tokens spent with a cap set outside your code. The next step in the series leaves text behind and moves on to images, in Image Generation: Stable Diffusion, DALL-E and Midjourney.