Game backend observability: measuring what the player feels
A game backend can have every dashboard green and still play badly. The HTTP error rate is low, the CPU is at 40%, and meanwhile some players see their character jerk because the server skipped ticks. The metrics of a web service (requests per second, errors, average latency) do not see this kind of problem. You need domain measurements: the server tick, the long tail of latency, and alerts that fire for what really consumes players' trust and not for every wobble.
The article tackles four decisions that are expensive when made badly, each with a listing run on synthetic data with a fixed seed: which labels to put on metrics, how to read the tick without being fooled by the average, why percentiles are approximated and how to choose buckets, and when an alert deserves to wake someone up. Prometheus, Grafana and an OpenTelemetry collector are not installed on this machine: where a fragment is meant for those systems I label it "sketch to adapt" and I report no output.
The tick as the server's unit of work is introduced in real-time state synchronisation; the latency part as seen by the player, with its budget, is in cloud gaming with WebRTC. The experiments and configurations you change in production, and which the metrics must be able to attribute, are in LiveOps with feature flags.
What to measure
A concrete example helps to choose: a player has a bad experience if they feel delayed commands (latency), see jerky movement (skipped ticks or packet loss), wait too long before playing (matchmaking queue) or abandon the match. Each sensation corresponds to a measurement.
- Network: RTT per connection and packet loss, as a histogram, to get the distribution and not the average
- Game loop: the duration of each tick and the number of ticks that exceed their own period
- Match: abandonments, duration, active matches
- Matchmaking: queue depth and wait time, this too as a histogram
- Capacity: connected players and busy servers, for sizing
None of these items has a universal "target" value here: the thresholds come from your game. A competitive shooter and a card game do not have the same tolerance for latency, and that is why the article reports no target tables. Where I need a number (a bucket, a threshold) I declare it as a choice of the design, to be changed with your data.
Labels decide how much the metric costs
Prometheus stores one time series for every combination of label values. Its best-practices documentation is clear: do not use labels for
high-cardinality dimensions, such as user identifiers. The reason is arithmetic, and the listing shows it: a classic histogram with 10 buckets creates
13 series per combination (10 buckets, the +Inf bucket, and the _sum and _count series), so putting player_id among the labels
multiplies the series by the number of players.
import random
# A metric produces one time series for every REAL combination of label values.
# A classic Prometheus histogram creates (buckets + 1 for +Inf + 2 for _sum and _count) per combination.
BUCKETS = 10
PER_HISTOGRAM = BUCKETS + 1 + 2
rng = random.Random(1)
players = [(f"player_{i}", rng.choice(["eu", "us", "asia", "sa"]), rng.choice(["pc", "console", "mobile"])) for i in range(20_000)]
per_player = len(set(players)) * PER_HISTOGRAM # labels: player_id, region, platform
per_region = len({(r, p) for _, r, p in players}) * PER_HISTOGRAM # labels: region, platform
per_match_gauge = 2_000 # a gauge labelled with match_id, 2,000 active matches
per_region_gauge = len({r for _, r, _ in players})
fmt = lambda n: f"{n:,}"
print(f"RTT histogram with player_id (20,000 online): {fmt(per_player)} series")
print(f"RTT histogram by region and platform: {fmt(per_region)} series")
print(f"tickrate gauge with match_id (2,000 matches): {fmt(per_match_gauge)} series")
print(f"tickrate gauge by region: {fmt(per_region_gauge)} series")
print(f"ratio between the two histogram versions: {per_player // per_region}x")
# Output:
# RTT histogram with player_id (20,000 online): 260,000 series
# RTT histogram by region and platform: 156 series
# tickrate gauge with match_id (2,000 matches): 2,000 series
# tickrate gauge by region: 4 series
# ratio between the two histogram versions: 1666xWith 20,000 players online, the per-player RTT histogram produces 260,000 series, against 156 if you aggregate by region and platform: over 1,600 times as many for information that no dashboard reads player by player. The same goes for the tickrate gauge with the match as a label: 2,000 series against 4. The single-player case is solved with logs and traces, which are designed for it, and metrics stay aggregates. The Go sketch that follows shows the correct version of the definitions: region and platform labels, seconds as the unit, and a bucket that coincides with the SLO threshold.
// Sketch to adapt, NOT run (Go is not installed): metrics with low-cardinality labels.
package metrics
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
var (
// Base unit: seconds. The 0.1 bucket coincides with the SLO threshold (100 ms).
PlayerRTT = promauto.NewHistogramVec(prometheus.HistogramOpts{
Namespace: "gameserver", Subsystem: "network", Name: "rtt_seconds",
Help: "Round-trip time measured by the server",
Buckets: []float64{0.01, 0.02, 0.04, 0.06, 0.08, 0.1, 0.15, 0.2, 0.3, 0.5},
}, []string{"region", "platform"}) // no player_id, no match_id
// How long processing a tick takes: the 0.015625 bucket is the period at 64 ticks/s.
TickSeconds = promauto.NewHistogramVec(prometheus.HistogramOpts{
Namespace: "gameserver", Subsystem: "loop", Name: "tick_seconds",
Help: "Processing time of a tick",
Buckets: []float64{0.002, 0.004, 0.008, 0.015625, 0.03125, 0.0625},
}, []string{"region", "mode"})
// Ticks that exceeded their own period: the signal the mean hides.
TickOverruns = promauto.NewCounterVec(prometheus.CounterOpts{
Namespace: "gameserver", Subsystem: "loop", Name: "tick_overruns_total",
Help: "Ticks over the period",
}, []string{"region", "mode"})
)
I could not compile it: Go is not installed here. The signatures (promauto.NewHistogramVec, HistogramOpts, CounterOpts) are those
of the Prometheus Go client; check them against your version. The Prometheus documentation also recommends base units, and for time they are seconds,
which is why the RTT here is in seconds and the SLO bucket is 0.1 and not 100.
The tick: the average hides the spikes
A server at 64 ticks per second has 15.6 ms to process each tick. If the average tick lasts 8 ms everything seems fine, but a tick that lasts 45 ms skips almost three periods, and the player sees it as a jerk. The listing simulates a minute of ticks with mostly low cost and 2% of spikes six times as costly (garbage collection, a crowded scene, a disk access), then measures the average, percentiles and effective tickrate of a loop that does not catch up on the delay.
import random, statistics
# A game loop at 64 ticks/s (period 15.625 ms). The cost of a tick is mostly low, with the occasional spike.
rng = random.Random(9)
PERIOD = 1000 / 64
def tick_cost():
base = rng.lognormvariate(1.9, 0.35) # ms: median ~6.7 ms
return base * (6 if rng.random() < 0.02 else 1) # 2% of ticks cost 6 times as much (GC, entity spike, I/O)
costs = [tick_cost() for _ in range(64 * 60)] # one minute of ticks
def late_loop(costs):
"""Loop with no catch-up: every tick takes at least one period; a slow one takes more and pushes back the following ones."""
t, ticks = 0.0, 0
for c in costs:
t += max(c, PERIOD) # a slow tick takes more than one period and the rest slips
if t > 60_000: break
ticks += 1
return ticks / 60
print(f"mean cost of a tick: {statistics.mean(costs):.1f} ms ({100*statistics.mean(costs)/PERIOD:.0f}% of the {PERIOD:.1f} ms period)")
qs = statistics.quantiles(costs, n=100)
print(f"p50 {qs[49]:.1f} ms, p99 {qs[98]:.1f} ms")
over = sum(1 for c in costs if c > PERIOD)
print(f"ticks over the period: {over} out of {len(costs)} ({100*over/len(costs):.1f}%)")
print(f"effective tickrate with a slipping loop: {late_loop(costs):.1f} ticks/s (target 64)")
# Output:
# mean cost of a tick: 8.0 ms (51% of the 15.6 ms period)
# p50 6.8 ms, p99 45.1 ms
# ticks over the period: 111 out of 3840 (2.9%)
# effective tickrate with a slipping loop: 61.3 ticks/s (target 64)
The average is 8 ms, 51% of the period; the p50 is 6.8 ms; the p99 rises to 45.1 ms. 2.9% of ticks (111 out of 3,840) exceed their own period, and the effective tickrate of the loop
is 61.3 ticks/s against the 64 expected. The figure to watch is therefore the fraction of late ticks and the tail of tick times, not the average. Note also
that an alert on the tickrate that fires below 90% of the target (57.6 ticks/s) would never fire here, even with three jerks every hundred ticks: that is why the Go sketch
has the counter tick_overruns_total next to the histogram. The numbers come from my model, not from a real server; the shape (good average, bad tail) is what to look for in your data.
Percentiles: approximations, and where to put the buckets
A Prometheus histogram stores only how many samples fall below each threshold. The percentile is estimated by histogram_quantile
by interpolating linearly inside the bucket where the rank falls; the documentation warns that the estimate is more accurate with denser buckets. The listing
implements that rule and compares it with the percentile computed on the raw data, using 50,000 synthetic RTTs with 1.5% of spikes between 150 and 400 ms.
import random, bisect
rng = random.Random(4)
# Synthetic RTTs in ms: most between 20 and 70, 1.5% are spikes between 150 and 400
rtts = [rng.lognormvariate(3.7, 0.35) if rng.random() > 0.015 else rng.uniform(150, 400) for _ in range(50_000)]
def to_histogram(values, bounds):
"""Cumulative counters as in Prometheus: for each 'le', how many samples are <= le."""
s = sorted(values)
return [(le, bisect.bisect_right(s, le)) for le in bounds] + [(float("inf"), len(s))]
def histogram_quantile(q, hist):
"""Same rule as Prometheus: find the bucket of the rank and interpolate linearly inside it."""
total = hist[-1][1]
rank = q * total
prev_le, prev_count = 0.0, 0
for le, count in hist:
if count >= rank:
if le == float("inf"):
return prev_le # beyond the last finite bucket, its upper bound is returned
return prev_le + (le - prev_le) * (rank - prev_count) / (count - prev_count)
prev_le, prev_count = le, count
true_p99 = sorted(rtts)[int(0.99 * len(rtts)) - 1]
FINE = [10, 20, 40, 60, 80, 100, 150, 200, 300, 500]
COARSE = [50, 100, 500]
print(f"true p99 on the raw samples: {true_p99:.1f} ms")
for name, bounds in (("10 buckets", FINE), ("3 buckets", COARSE)):
h = to_histogram(rtts, bounds)
print(f"{name}: estimated p99 {histogram_quantile(0.99, h):.1f} ms")
ok = dict(h)[100] / len(rtts)
print(f" share with RTT <= 100 ms, read from bucket le=100: {100*ok:.2f}%")
exact = sum(1 for v in rtts if v <= 100) / len(rtts)
print(f"share with RTT <= 100 ms on the raw samples: {100*exact:.2f}%")
# Output:
# true p99 on the raw samples: 231.8 ms
# 10 buckets: estimated p99 232.8 ms
# share with RTT <= 100 ms, read from bucket le=100: 98.03%
# 3 buckets: estimated p99 297.2 ms
# share with RTT <= 100 ms, read from bucket le=100: 98.03%
# share with RTT <= 100 ms on the raw samples: 98.03%The true p99 is 231.8 ms. With the 10 buckets of our sketch the estimate is 232.8 ms; with only 3 buckets (50, 100, 500) it rises to 297.2 ms, an error of 65 ms caused only by the choice of boundaries. The share of requests under 100 ms, instead, is exact in both cases (98.03%), because 100 is a bucket boundary. A design rule follows: put a bucket exactly on the threshold of your SLO, and every count on that threshold will be exact. It is what the Prometheus documentation suggests, when it says that if you know an SLO in advance you can use fixed boundaries to get an accurate calculation.
Alerts: when to wake someone up
A fixed-threshold alert ("errors above 0.5%") fires even for a very brief spike that does not significantly dent the month. The alternative described in Google's SRE workbook is to watch the speed at which the error budget is consumed (burn rate). With a 99.5% SLO the budget is 0.5% of the month's requests; a burn rate of 14.4 over one hour consumes 2% of the monthly budget, and it is the threshold the workbook associates with a page. To avoid false alarms, the rule requires the 5-minute window to be above the threshold too, so the alert switches off as soon as the problem ends.
import random
rng = random.Random(12)
SLO, MIN_MONTH = 0.995, 30 * 24 * 60
BUDGET = 1 - SLO # allowed share of errors: 0.5%
# 24 hours of error ratio per minute: background noise + three events
err = [max(0.0, rng.gauss(0.0015, 0.0020)) for _ in range(1440)]
def add(start, minutes, ratio):
for m in range(start, start + minutes): err[m] = ratio
add(200, 3, 0.012) # B: 3-minute spike at 1.2%
add(600, 20, 0.40) # A: a real incident, 20 minutes with 40% of requests failing
add(900, 360, 0.008) # C: slow burn, 6 hours at 0.8%
def avg(series, end, minutes):
w = series[max(0, end - minutes + 1): end + 1]
return sum(w) / len(w)
def firing_periods(rule):
out, active = [], False
for m in range(1440):
on = rule(m)
if on and not active: out.append(m)
active = on
return out
# Rule 1: fixed threshold. Average over the last 5 minutes above the budget (0.5%), for at least 2 consecutive minutes
def threshold(m): return m >= 1 and avg(err, m, 5) > BUDGET and avg(err, m - 1, 5) > BUDGET
# Rule 2: multi-window burn rate (SRE Workbook): 14.4 times the consumption rate, over 1 hour AND over 5 minutes
def burn(m): return avg(err, m, 60) > 14.4 * BUDGET and avg(err, m, 5) > 14.4 * BUDGET
def consumed(start, minutes, ratio): return 100 * ratio * minutes / (BUDGET * MIN_MONTH)
print(f"monthly budget: {BUDGET * MIN_MONTH:.0f} equivalent minutes of total failure")
for name, s, d, r in (("B 3 min spike at 1.2%", 200, 3, 0.012), ("A incident 20 min at 40%", 600, 20, 0.40), ("C slow 6 h at 0.8%", 900, 360, 0.008)):
print(f" event {name}: consumes {consumed(s, d, r):.2f}% of the monthly budget")
for name, rule in (("fixed threshold 5m > 0.5% for 2m", threshold), ("burn rate 14.4x over 1h and 5m", burn)):
f = firing_periods(rule)
print(f"{name}: {len(f)} alerts, starting at minutes {f}")
a = [m for m in f if 600 <= m < 640]
print(f" detects A? {'yes, after ' + str(a[0] - 600) + ' min' if a else 'no'}; detects B? {'yes' if any(200 <= m < 215 for m in f) else 'no'}; detects C? {'yes' if any(900 <= m < 1260 for m in f) else 'no'}")
# Output:
# monthly budget: 216 equivalent minutes of total failure
# event B 3 min spike at 1.2%: consumes 0.02% of the monthly budget
# event A incident 20 min at 40%: consumes 3.70% of the monthly budget
# event C slow 6 h at 0.8%: consumes 1.33% of the monthly budget
# fixed threshold 5m > 0.5% for 2m: 3 alerts, starting at minutes [202, 601, 903]
# detects A? yes, after 1 min; detects B? yes; detects C? yes
# burn rate 14.4x over 1h and 5m: 1 alerts, starting at minutes [610]
# detects A? yes, after 10 min; detects B? no; detects C? noEvent B is a 3-minute spike at 1.2% that consumes 0.02% of the monthly budget: the fixed threshold flags it (alert at minute 202), the burn rate does not, and that is right. Event A is a real incident, 20 minutes with 40% of requests failing, 3.70% of the budget: both see it, but the threshold after 1 minute and the burn rate after 10, and that is the price to pay. Event C is a slow burn (6 hours at 0.8%, 1.33% of the budget): the fixed threshold flags it, the fast burn rate does not, because it is made for fast consumption. For slow burns the workbook provides a separate rule on long windows (10% of the budget in 3 days, burn rate 1) that opens a ticket and wakes nobody.
# Sketch to adapt: YAML syntax verified, rules NOT evaluated by Prometheus (promtool is not installed).
groups:
- name: rtt-slo
rules:
- record: slo:rtt_over_100ms:ratio_rate5m
expr: 1 - sum(rate(gameserver_network_rtt_seconds_bucket{le="0.1"}[5m])) / sum(rate(gameserver_network_rtt_seconds_count[5m]))
- record: slo:rtt_over_100ms:ratio_rate1h
expr: 1 - sum(rate(gameserver_network_rtt_seconds_bucket{le="0.1"}[1h])) / sum(rate(gameserver_network_rtt_seconds_count[1h]))
# SLO 99.5% -> budget 0.005. Consumption 14.4 times faster than the sustainable rate, over 1 hour AND over 5 minutes.
- alert: RttBudgetBurnFast
expr: slo:rtt_over_100ms:ratio_rate1h > (14.4 * 0.005) and slo:rtt_over_100ms:ratio_rate5m > (14.4 * 0.005)
labels:
severity: page
annotations:
summary: "The RTT SLO budget is being consumed too fast"
The two record rules precompute the fraction of requests over 100 ms over 5 minutes and one hour, using exactly the le="0.1" bucket defined earlier.
The alert fires only if both exceed 14.4 times the budget (0.005). The file is valid YAML (I read it with a parser), but the rules were not evaluated by Prometheus,
because promtool is not installed: try them on real data with promtool test rules before entrusting a page to them.
Traces: sampling without losing the errors
A distributed trace follows a request from one service to another, useful to understand why a matchmaking takes 8 seconds instead of 2. Saving all the traces costs too much, so you sample. The OpenTelemetry documentation distinguishes head sampling, where the decision is made as early as possible and does not look at the trace as a whole, from tail sampling, where the decision is made once the trace is complete. With a fixed-probability sampler (for example 10% of trace identifiers) you cannot guarantee keeping the traces with errors: at decision time it is not yet known that there will be an error. If you want all the slow or failed traces, tail sampling in the collector is the right tool, at the price of letting traces flow through and be held in memory until the decision.
An experience score: useful, but watch the average
For management dashboards a single number is often wanted: an experience score from 0 to 100 combining RTT, tickrate and packet loss. It is a good tool on two conditions: each component is normalised explicitly, and the combination does not hide a serious fault. The listing compares a weighted average with the minimum of the components. Thresholds and weights are the team's choices: the values in the code are examples, to be validated by comparing the score with real abandonments.
def lin(x, good, bad):
"""Score 100 if x <= good, 0 if x >= bad, linear in between (also works if good > bad: then higher is better)."""
if good == bad: return 100.0
return max(0.0, min(100.0, 100 * (x - bad) / (good - bad)))
def components(rtt_ms, tick_ratio, loss_pct):
return {
"rtt": lin(rtt_ms, good=40, bad=150), # thresholds chosen by the team, to be validated on your own game
"tick": lin(tick_ratio, good=0.95, bad=0.70),
"loss": lin(loss_pct, good=0, bad=2),
}
WEIGHTS = {"rtt": 0.45, "tick": 0.35, "loss": 0.20}
def pes_mean(c): return sum(WEIGHTS[k] * c[k] for k in c)
def pes_min(c): return min(c.values())
cases = {
"all good": (35, 0.99, 0.0),
"mediocre network": (95, 0.97, 0.5),
"server at 50% tickrate": (35, 0.50, 0.0),
"network and server at limit": (110, 0.80, 1.2),
}
print(f"{'case':<28}{'weighted mean':>14}{'minimum':>8}")
for name, args in cases.items():
c = components(*args)
print(f"{name:<28}{pes_mean(c):>14.1f}{pes_min(c):>8.1f}")
# Output:
# case weighted mean minimum
# all good 100.0 100.0
# mediocre network 72.5 50.0
# server at 50% tickrate 65.0 0.0
# network and server at limit 38.4 36.4The "server at half the tickrate" case is the lesson: with perfect RTT and loss, the weighted mean gives 65.0, a mark that looks "acceptable", while the tickrate is halved and the minimum of the components says 0. An average is convenient for trends but can hide a broken component; a minimum (or an average with a cap on the worst component) brings it out. Always show the score together with its three components, and do not publish tables of the kind "score 60-74 equals 5-10% abandonments" without having measured them on your game.
When you don't need all this
A studio with a single game server and a few hundred players can start from a structured log per match with RTT and tick duration, and from a script that summarises it. Prometheus, traces and burn rate pay off when you have more services, more regions and someone on call whose sleep a wrong alert costs.
Frequently asked questions
Can I use player_id as a label of a Prometheus metric?
No: the documentation advises against high-cardinality labels such as user identifiers. In the article's example the RTT histogram goes from 156 to 260,000 series with 20,000 players. For the single player use logs and traces.
How do you choose an alert for a latency SLO?
With the burn rate over two windows, as in Google's SRE workbook: consumption at 14.4 times the sustainable rate over one hour and over five minutes for pages, and long windows for tickets. It detects less noise than a fixed threshold, but with a few minutes of delay.
Why doesn't Prometheus's p99 match the true one?
Because histogram_quantile interpolates inside the bucket: the sparser the buckets, the further the estimate drifts. In the example, 3 buckets give 297 ms against a true 232 ms. Put a bucket exactly on the SLO threshold to get exact counts.
What you can do now
You can choose a metric's labels by counting the series they produce, read the tick from the tail of the times and the fraction of late ticks and not from the average, and place the buckets on the SLO threshold. You can set an alert on the consumption of the error budget, you know how much detection delay it brings compared with a fixed threshold, and why fixed-probability trace sampling loses the errors. Still to be fixed, on your data: the thresholds, the score weights and the real distribution of tick times.
Sources
- Prometheus, Metric and label naming - the ban on high-cardinality labels such as user identifiers, and base units, consulted on 2026-11-12
- Prometheus, Histograms and summaries - the interpolation of histogram_quantile and fixed boundaries for a known SLO, consulted on 2026-11-12
- Google SRE Workbook, Alerting on SLOs - burn rate, multiple windows and the 14.4 / 6 / 1 table, consulted on 2026-11-12
- OpenTelemetry, Sampling - head and tail sampling, consulted on 2026-11-12
The Python listings are run with Python 3 on synthetic data with a fixed seed: the counts and percentages describe the model, not a real game. The Go sketch and the YAML rules were not run (Go and promtool are not installed); the YAML file was only read with a parser.
Commenti
Caricamento commenti...
Sign in to comment