07 - Live Operations (LiveOps): Event Systems & Feature Flags

LiveOps: changing a game in production without releasing a new client

A game that lives on seasonal events has a problem that "closed" games do not have: the content changes more often than the client. The client lives in the stores, and every new version goes through a review and an update that half of the players install late. The backend, on the other hand, is yours to control. The answer is to move to the server the decision of what is active, for whom and from when, and leave the client the job of carrying it out. This article puts together the four mechanisms you need, and a fifth that whoever builds them discovers last: how to go back quickly when the event goes wrong.

The mechanisms are feature flags (switches evaluated on the server, even for only a part of the players), A/B experiments (a measured comparison between two versions), the server-driven interface (the server describes the screens, the client draws them) and rollback. Every section has a listing that I ran with Node 24: they are single-process models, with a fixed seed, that prove the logic without network or database. Where a number comes out of a simulation I say so, and I do not present it as a measurement on a real game.

Prerequisite: knowing where player profiles and ratings live, described in Anatomy of a multiplayer game backend. The previous part of the series, on the open-source Nakama framework, is Open Match and Nakama.

What a live event contains

An event is not a banner with a countdown. It touches the shop, the missions, the game mode and the interface, and for this reason it is best to make it a single record that the other services read, instead of a configuration scattered in five places. A concrete example: "Halloween Horror Night" from 24 to 31 October for European players whose account is more than seven days old.

• Window: start and end in UTC, never in the server's local time
• Audience: region, platform, player segment, account age
• Linked flags: the switches that turn on the map, the themed currency, the shop
• Exposure: the share of players who see it, for a gradual launch
• Way back: which flags to turn off and what to do with the data already written

The last item is the one missing from improvised projects. An event that gives away currency or items writes into the players' data, and turning off the flag does not erase what has already been granted: it must be decided before launch whether that data stays, is compensated or is restored from a copy. The first four items are data; the fifth is a decision, and it is written by whoever knows the game's economy.

Feature flags: evaluating per player, in a stable way

A feature flag is a function that, given a switch and a player, answers yes or no. The delicate part is the percentage: "on for 10%" must always mean the same players, otherwise someone sees the Halloween map on first login and does not find it on the second. The solution is a deterministic draw: you compute a hash of the player's identifier and reduce it to a number from 0 to 99, the "bucket". Whoever has a bucket below the percentage sees the feature.

In the listing the choice falls on 32-bit FNV-1a, with Math.imul for the 32-bit multiplication without slipping into floating point. The line that matters is bucket(playerId, salt): the salt is the flag's key, and it is there so that two different flags are not switched on for the same people. If two flags at 10% shared the salt, the lucky 10% would always get everything, and the other 90% would never see anything. The rules are ANDed, with early exit at the first false, and the override list for QA and beta testers takes precedence over everything else. Pete Hodgson, in the article hosted by Martin Fowler, classifies these switches by purpose (release, experiment, operational, permissioning), and this is useful because they have different lifetimes and owners.

// 32-bit FNV-1a: fast, dependency-free, spreads identifiers well
function fnv1a(s: string): number {
  let h = 0x811c9dc5;
  for (let i = 0; i < s.length; i++) { h ^= s.charCodeAt(i); h = Math.imul(h, 0x01000193); }
  return h >>> 0;
}
// The salt (here the flag key) makes the draws of different flags independent
function bucket(playerId: string, salt: string): number { return fnv1a(salt + ':' + playerId) % 100; }

function evaluateFlag(flag: FeatureFlag, p: PlayerContext, nowMs: number): boolean {
  if (!flag.enabled) return false;
  if (flag.overrideUsers?.includes(p.id)) return true;
  if (flag.countries && !flag.countries.includes(p.country)) return false;
  if (flag.platforms && !flag.platforms.includes(p.platform)) return false;
  if (flag.segments && !flag.segments.some(s => p.segments.includes(s))) return false;
  if (flag.minAccountAgeDays !== undefined && (nowMs - p.createdAtMs) / 86_400_000 < flag.minAccountAgeDays) return false;
  if (flag.percentage !== undefined && bucket(p.id, flag.key) >= flag.percentage) return false;
  return true;
}

The same file, with a test on 100,000 synthetic identifiers, shows what you can promise. percentage: 10 switches the flag on for 10,024 players, that is 10.02%: the number is not exact because the hash is a pseudo-random function, not a counter. The result is identical an hour later. With a different salt the two 10% flags share 1,037 players, close to the roughly 1,000 expected if they were independent (with the same salt the two sets would coincide by construction). The distribution of buckets ranges between 947 and 1,052 players per bucket against the ideal 1,000.

// --- test ---
const NOW = Date.UTC(2026, 9, 31, 18, 0, 0);
const players: PlayerContext[] = Array.from({ length: 100_000 }, (_, i) => ({
  id: 'p' + String(i).padStart(6, '0'), country: i % 4 === 0 ? 'IT' : 'DE',
  platform: 'pc', segments: [], createdAtMs: NOW - 40 * 86_400_000,
}));
const halloween: FeatureFlag = { key: 'halloween_map', enabled: true, percentage: 10 };
const pumpkin: FeatureFlag = { key: 'pumpkin_currency', enabled: true, percentage: 10 };

const on = players.filter(p => evaluateFlag(halloween, p, NOW));
console.log('with percentage=10:', on.length, 'players out of', players.length);
const stable = players.every(p => evaluateFlag(halloween, p, NOW) === evaluateFlag(halloween, p, NOW + 3_600_000));
console.log('same outcome one hour later:', stable);
const both = players.filter(p => evaluateFlag(halloween, p, NOW) && evaluateFlag(pumpkin, p, NOW)).length;
console.log('in both flags (different salt):', both, '(expected if independent: about 1000)');
const onlyIt = { key: 'it_only', enabled: true, countries: ['IT'] };
console.log('Italy only:', players.filter(p => evaluateFlag(onlyIt, p, NOW)).length);

const counts = new Array(100).fill(0);
for (const p of players) counts[bucket(p.id, 'halloween_map')]++;
console.log('buckets 0..99 over 100000 ids: min', Math.min(...counts), 'max', Math.max(...counts), '(uniform: 1000)');

// Output:
// with percentage=10: 10024 players out of 100000
// same outcome one hour later: true
// in both flags (different salt): 1037 (expected if independent: about 1000)
// Italy only: 25000
// buckets 0..99 over 100000 ids: min 947 max 1052 (uniform: 1000)

When a flag is the wrong choice

A flag left on forever becomes a dead branch that nobody dares remove. The same article distinguishes transient flags, to be deleted as soon as the event ends, from a few long-lived operational switches (the "kill switches", which degrade non-vital functions when the system is under load). Every flag should be born with an owner and a date on which to remove it.

Switching off fast: how long a rollback really takes

The control panel writes "enabled: false", but the backend nodes read the flags from a cache so as not to query the database on every request. The rollback time is therefore the cache's lifetime in the worst case. A TTL of 30 seconds means that for almost half a minute some of the nodes keep serving the event. There are two remedies, and the second is the one people forget.

The first is active invalidation: whoever turns off the flag publishes a message and the nodes empty their cache. But the Redis documentation says that Pub/Sub has at-most-once delivery semantics: a message is delivered once, if all goes well, and there is no retry. A node that reconnects at that moment will never receive it. The second remedy is a version number in the store, which every node checks at short intervals: cheap, and it does not depend on a message that can get lost. The listing simulates three strategies and measures, in simulated time, how long passes before all the nodes are off.

// Teaching model: how long does a rollback last? Simulated time in seconds, no network.
interface Store { flags: Record<string, boolean>; version: number; }

class FlagNode {
  private cache: Record<string, boolean> = {};
  private cachedAt = -Infinity;
  private lastPoll = -Infinity;
  private seenVersion = -1;
  private store: Store;
  private ttlSec: number;
  private pollSec: number | null;
  constructor(store: Store, ttlSec: number, pollSec: number | null) {
    this.store = store; this.ttlSec = ttlSec; this.pollSec = pollSec;
  }
  // Reads from the cache. Re-reads the store if the TTL has expired, or, when the
  // version check is active, if every pollSec seconds it discovers a new version.
  isOn(flag: string, now: number): boolean {
    let refresh = now - this.cachedAt >= this.ttlSec;
    if (!refresh && this.pollSec !== null && now - this.lastPoll >= this.pollSec) {
      this.lastPoll = now;
      refresh = this.store.version !== this.seenVersion;
    }
    if (refresh) { this.cache = { ...this.store.flags }; this.cachedAt = now; this.seenVersion = this.store.version; }
    return this.cache[flag];
  }
  invalidate(): void { this.cachedAt = -Infinity; }
}

type Strategy = 'TTL only' | 'TTL + delivered invalidation' | 'TTL + lost invalidation + version';
function secondsUntilAllOff(strategy: Strategy, rollbackAt: number): number {
  const store: Store = { flags: { halloween_map: true }, version: 1 };
  const poll = strategy.endsWith('version') ? 5 : null;
  const nodes = [new FlagNode(store, 30, poll), new FlagNode(store, 30, poll)];
  for (let t = 0; t < rollbackAt; t++) nodes.forEach(n => n.isOn('halloween_map', t)); // warm cache
  store.flags.halloween_map = false; store.version++;                                  // the operator turns the flag off
  if (strategy === 'TTL + delivered invalidation') nodes.forEach(n => n.invalidate());
  if (strategy === 'TTL + lost invalidation + version') nodes[0].invalidate();      // node 1 does not receive the message
  for (let t = rollbackAt; t < rollbackAt + 120; t++) {
    if (nodes.every(n => !n.isOn('halloween_map', t))) return t - rollbackAt;
  }
  return -1;
}
const strategies: Strategy[] = ['TTL only', 'TTL + delivered invalidation', 'TTL + lost invalidation + version'];
for (const s of strategies) {
  // the rollback moment falls at every point of the cache's 30 s cycle: take the worst case
  const waits = Array.from({ length: 30 }, (_, k) => secondsUntilAllOff(s, 100 + k));
  console.log(s.padEnd(38), '-> worst:', Math.max(...waits), 's, best:', Math.min(...waits), 's');
}

// Output:
// TTL only                               -> worst: 29 s, best: 0 s
// TTL + delivered invalidation           -> worst: 0 s, best: 0 s
// TTL + lost invalidation + version      -> worst: 4 s, best: 0 s

With the TTL alone the worst case is 29 seconds, one less than the TTL, and the best is zero: it depends on the point of the cycle at which the switch-off falls. Delivered invalidation brings it to zero. The interesting case is the third: the message reaches only node 0 and node 1 loses it, but the version check every 5 seconds closes the gap anyway within 4 seconds. These are simulated seconds on a two-node model: real numbers depend on the network and the load, but the reasoning remains valid. The combination is therefore TTL as a safety net, invalidation for speed and version for correctness.

Soft, hard and kill switch

Distinguishing the rollback levels avoids using the biggest weapon for a small problem. The soft rollback turns off the flags and does not touch the data: it is reversible and is done first. The hard one restores the economy data from a copy, is slow, needs an operator and should be done only if the data has been corrupted. The kill switch stops traffic to an entire service and is reserved for serious failures. The time for the hard restore depends on the size of the economy and on the copy: it is measured in a dress rehearsal, not assumed.

A/B experiments: measuring without fooling yourself

An experiment assigns each player to a variant with the same mechanism as flags (a stable hash of the player and of the experiment), records a metric and then compares. The most common comparison in games is between two percentages, for example the conversion to the seasonal pass. The test used is the z test for two proportions, with the pooled proportion in the denominator: it is the formula of the NIST Engineering Statistics Handbook, and the listing implements it. Before launching, however, you need a number that is computed at the desk: how many players per arm.

// Cumulative distribution function of the standard normal (Abramowitz-Stegun 7.1.26, error < 1.5e-7)
function normalCdf(x: number): number {
  const z = Math.abs(x) / Math.SQRT2, t = 1 / (1 + 0.3275911 * z);
  const erf = 1 - (((((1.061405429 * t - 1.453152027) * t) + 1.421413741) * t - 0.284496736) * t + 0.254829592) * t * Math.exp(-z * z);
  return x >= 0 ? 0.5 * (1 + erf) : 0.5 * (1 - erf);
}
// Two-tailed z test for two proportions (pooled proportion formula)
function zTest(x1: number, n1: number, x2: number, n2: number): { z: number; p: number } {
  const p1 = x1 / n1, p2 = x2 / n2, pooled = (x1 + x2) / (n1 + n2);
  const se = Math.sqrt(pooled * (1 - pooled) * (1 / n1 + 1 / n2));
  const z = (p1 - p2) / se;
  return { z, p: 2 * (1 - normalCdf(Math.abs(z))) };
}
// Users per arm to detect p0 -> p1 with two-tailed alpha 5% and power 80% (z = 1.96 and 0.84)
function sampleSizePerArm(p0: number, p1: number): number {
  const pbar = (p0 + p1) / 2;
  const num = 1.96 * Math.sqrt(2 * pbar * (1 - pbar)) + 0.8416 * Math.sqrt(p0 * (1 - p0) + p1 * (1 - p1));
  return Math.ceil((num * num) / ((p1 - p0) ** 2));
}

The sampleSizePerArm function answers the question "how many users do I need to tell 10% from 12% with 5% significance and 80% power?". The answer is 3,841 per arm, and the next experiment uses exactly that number. Reducing the sample to "finish sooner" is not free: it lowers the power, that is the probability of noticing an effect that is there. With samples that are too small, an experiment that "shows no differences" often cannot show them.

const n = sampleSizePerArm(0.10, 0.12);
console.log('users per arm to see 10% -> 12%:', n);

// 1) A real experiment: control 10%, variant 12%, exactly n users per arm
const r = rng(42);
const run = (p: number, k: number) => { let x = 0; for (let i = 0; i < k; i++) if (r() < p) x++; return x; };
const xc = run(0.10, n), xv = run(0.12, n);
const res = zTest(xv, n, xc, n);
console.log(`control ${xc}/${n}, variant ${xv}/${n}, z=${res.z.toFixed(2)}, p=${res.p.toFixed(4)}`);

// 2) A/A test (no real difference): how many false positives with 14 days of data?
const EXPS = 1000, DAYS = 14, PER_DAY = 500, ALPHA = 0.05;
let endOnly = 0, peeking = 0;
for (let e = 0; e < EXPS; e++) {
  let xa = 0, xb = 0, na = 0, nb = 0, stopped = false;
  for (let d = 1; d <= DAYS; d++) {
    for (let i = 0; i < PER_DAY; i++) { if (r() < 0.10) xa++; if (r() < 0.10) xb++; }
    na += PER_DAY; nb += PER_DAY;
    if (!stopped && zTest(xa, na, xb, nb).p < ALPHA) { stopped = true; }   // look every day and stop at the first "significant" result
    if (d === DAYS && zTest(xa, na, xb, nb).p < ALPHA) endOnly++;
  }
  if (stopped) peeking++;
}
console.log(`A/A, ${EXPS} experiments: false positives looking only at day ${DAYS}: ${(100 * endOnly / EXPS).toFixed(1)}%`);
console.log(`A/A, ${EXPS} experiments: false positives looking every day and stopping at the first p<0.05: ${(100 * peeking / EXPS).toFixed(1)}%`);

// Output:
// users per arm to see 10% -> 12%: 3841
// control 383/3841, variant 495/3841, z=4.02, p=0.0001
// A/A, 1000 experiments: false positives looking only at day 14: 6.0%
// A/A, 1000 experiments: false positives looking every day and stopping at the first p<0.05: 22.9%

The first experiment, with a fixed seed, sees 383 conversions out of 3,841 in the control and 495 in the variant: z=4.02, p=0.0001. The second part is the most useful lesson. In an A/A test the two variants are identical, so every "win" is a false positive. Looking at the data only once, on day 14, the false positives are 6.0% over 1,000 simulated experiments, close to the expected 5%. Looking every day and stopping as soon as the p-value drops below 0.05 they rise to 22.9%. The topic is well known in the literature: Johari, Pekelis and Walsh show that classical p-values do not hold up under continuous monitoring and propose "always valid" methods. In practice: fix duration and sample beforehand and do not stop the experiment at the first peak, or use a sequential method designed for it.

Server-driven interface

In the server-driven interface the API response contains not data but screens: a list of sections, each with a type, a priority and a configuration. The client has a renderer for each type it knows. The advantage is changing the shop and the home without going through the stores; the cost is that client and server must agree on a vocabulary of types, and that the vocabulary grows over time, while old clients remain in circulation.

interface Section { id: string; type: string; priority: number; config: Record<string, unknown>; }
interface HomeScreen { version: number; sections: Section[]; }

const renderers: Record<string, (c: Record<string, unknown>) => string> = {
  banner: c => `[banner] ${c.title} - ${c.subtitle}`,
  shop_carousel: c => `[shop] ${c.title} (${(c.items as unknown[]).length} items)`,
};
function render(screen: HomeScreen): string[] {
  return [...screen.sections]
    .sort((a, b) => a.priority - b.priority)
    .map(s => (renderers[s.type] ? renderers[s.type](s.config) : null))
    .filter((line): line is string => line !== null); // unknown type: section skipped, no crash
}
const screen: HomeScreen = { version: 7, sections: [
  { id: 's2', type: 'shop_carousel', priority: 2, config: { title: 'Halloween Shop', items: [{ id: 'skin_1' }, { id: 'emote_1' }] } },
  { id: 's1', type: 'banner', priority: 1, config: { title: 'Halloween Horror Night', subtitle: 'ends in 3 days' } },
  { id: 's3', type: 'hologram_wall', priority: 3, config: {} },   // type introduced by a newer client
] };
console.log(render(screen));


// Output:
// [
//   '[banner] Halloween Horror Night - ends in 3 days',
//   '[shop] Halloween Shop (2 items)'
// ]

The line that protects the players is the final filter: the third section has type hologram_wall, introduced by a newer client, and it does not exist in the renderer. It is skipped, and the home shows the banner and the shop in order of priority. Without that rule an unknown type brings down the whole screen. The version field is there to cache the response and to know which vocabulary the client has seen. When not to use it: screens that need animations and complex interactions stay simpler and smoother in the client; the declarative interface is worth it for the shop, banners and lists.

Rewards: a retry must not pay twice

An event is, in large part, a system that assigns rewards. The client calls, the network times out, the client retries: if the server has already credited the first call, the second multiplies the prize. The defence is an idempotency key that identifies the operation (player, event, challenge) and not the attempt.

const wallet = new Map<string, number>();
const done = new Set<string>();
function grantReward(playerId: string, rewardKey: string, coins: number): 'credited' | 'already credited' {
  const idem = playerId + ':' + rewardKey;           // the key identifies the operation, not the attempt
  if (done.has(idem)) return 'already credited';
  wallet.set(playerId, (wallet.get(playerId) ?? 0) + coins);
  done.add(idem);
  return 'credited';
}
console.log(grantReward('p1', 'halloween-2026:challenge-1', 100));
console.log(grantReward('p1', 'halloween-2026:challenge-1', 100)); // retry after a timeout
console.log('p1 balance:', wallet.get('p1'));

// Output:
// credited
// already credited
// p1 balance: 100

The second call answers "already credited" and the balance stays at 100. The listing uses an in-memory set, which is fine for showing the idea and not for a real system: between two processes, the check followed by the write has a window in which both pass. In production the guarantee must live in the database, with a uniqueness constraint on the key, or in a transaction that writes key and balance together. If you skip this point, a double click or a client retry turns into inflated currency, which then nobody knows how to take back.

Before launch

• Every part of the event behind a flag that an operator can turn off without developer intervention
• Rollback rehearsed on staging, with the time measured and written in the runbook
• Load test at the expected peak, because the opening of the event concentrates requests
• Dates only on the server: a date written in the client cannot be extended without an update
• Flag change log: who, when and why, to understand a regression afterwards

These are five habits with the same root: reducing the number of things that only a developer can do during an incident. If you must pick just one to start with, pick the second: a rollback never rehearsed is a hope. Load tests and reading production metrics are in the article Observability of a game backend, while the analysis of event data is in the article on telemetry.

Frequently asked questions

What you can do now

You can describe an event as a record with a way back, evaluate a flag in a stable way that is independent of the others, estimate how long a rollback lasts and which combination of cache, message and version shortens it. You can size an experiment, and know why it should not be stopped at the first favourable result. You can have the client draw screens it does not know without letting it fall over, and protect rewards from retries.