09 - Cloud Gaming: Streaming with WebRTC and Edge Nodes

Cloud gaming: the input-to-image loop has a budget, and physics fixes part of it

In a game played from the cloud, the player presses a key, the command reaches a server, the server runs the game, draws a frame, compresses it and sends it back; the device decodes it and shows it. All of this has to happen fast enough that nobody notices. There is no universal threshold in milliseconds: you pick one for your kind of game (a shooter tolerates less than a strategy game) and treat it as a budget to be split among the stages of the loop. One piece of that budget is not yours to choose: the speed of propagation in fibre puts a ceiling on the distance between player and server.

The article starts from that ceiling, with a calculation you can run, and then follows the loop: WebRTC as the transport, the input channel and its sequence number, video encoding on the server, the choice of the edge node, quality control and GPU sharing. The listings are TypeScript run with Node 24 on synthetic data with a fixed seed; the browser fragment was only compiled. I have neither a game server nor a GPU here, so no number in this article is the measurement of a real service.

The path of packets to a dedicated game server is covered in the article on dedicated game servers and orchestration; client-side state synchronisation is in state synchronisation and rollback.

The latency budget and the limit of fibre

The loop has eight stages: reading the input, network to the server, game logic, rendering, encoding, network back, decoding and display. Six you govern with code and the choice of hardware. Two, the network legs, have a physical minimum: light in fibre travels at about c divided by the refractive index, which in glass is typically 1.47 (a generic value, not that of a specific cable). The listing computes the theoretical minimum RTT between two points from the straight-line distance, and then inverts the problem: given your target and the local stages you have measured, how much distance is left?

interface Place { name: string; lat: number; lon: number; }
const toRad = (d: number) => (d * Math.PI) / 180;
function haversineKm(a: Place, b: Place): number {
  const dLat = toRad(b.lat - a.lat), dLon = toRad(b.lon - a.lon);
  const h = Math.sin(dLat / 2) ** 2 + Math.cos(toRad(a.lat)) * Math.cos(toRad(b.lat)) * Math.sin(dLon / 2) ** 2;
  return 2 * 6371 * Math.asin(Math.sqrt(h));
}
// Physical limit: light in fibre travels at c/n. n = 1.47 is a typical value for glass, not for a specific cable.
const C_KM_S = 299_792.458, N_FIBER = 1.47;
const minRttMs = (km: number) => (2 * km) / (C_KM_S / N_FIBER) * 1000;
const maxKmForRtt = (rttMs: number) => ((rttMs / 1000) / 2) * (C_KM_S / N_FIBER);

const milan: Place = { name: 'Milan', lat: 45.4642, lon: 9.19 };
for (const dc of [
  { name: 'Frankfurt', lat: 50.1109, lon: 8.6821 },
  { name: 'Ashburn (Virginia)', lat: 39.0438, lon: -77.4874 },
]) {
  const km = haversineKm(milan, dc);
  console.log(`Milan - ${dc.name}: ${km.toFixed(0)} km as the crow flies, theoretical minimum RTT ${minRttMs(km).toFixed(1)} ms`);
}

// How much is left for the network? The stages are values you MEASURE on your own system (here: working assumptions).
const target = 100;
const stages: Record<string, number> = { input: 4, game: 8, render: 17, encode: 8, decode: 6, display: 10 };
const used = Object.values(stages).reduce((a, b) => a + b, 0);
const rttBudget = target - used;
console.log(`target ${target} ms, local stages ${used} ms, left for the network (RTT) ${rttBudget} ms`);
console.log(`with that RTT, even on an ideal straight path, the server can be at most ${maxKmForRtt(rttBudget).toFixed(0)} km away`);
console.log(`at 60 fps one frame lasts ${(1000 / 60).toFixed(1)} ms: rendering alone consumes one`);

// Output:
// Milan - Frankfurt: 518 km as the crow flies, theoretical minimum RTT 5.1 ms
// Milan - Ashburn (Virginia): 6814 km as the crow flies, theoretical minimum RTT 66.8 ms
// target 100 ms, local stages 53 ms, left for the network (RTT) 47 ms
// with that RTT, even on an ideal straight path, the server can be at most 4793 km away
// at 60 fps one frame lasts 16.7 ms: rendering alone consumes one

From Milan to Frankfurt the theoretical minimum RTT is 5.1 ms over 518 km; from Milan to Virginia it is 66.8 ms over 6,814 km. A real path is longer than the straight line and goes through equipment that adds time, so the real RTT will be higher. The local stages in stages are my assumptions, to be replaced with your measured values: with 53 ms used out of a 100 ms target, 47 ms are left for the network, and even on the ideal path the server cannot be farther than about 4,800 km. With the target and the stages of this example, a server in Virginia, for a player in Milan, is out of budget on physics alone, no matter how good the code is. At 60 fps, finally, one frame lasts 16.7 ms: rendering alone consumes one.

• Measurable with code: input reading, logic, rendering, encoding, decoding
• Dependent on the player's hardware: decoding and the screen's display time
• Dependent on location: the two network legs, with the physical minimum from the listing

The breakdown serves to decide where to invest. An encoder that is 3 ms faster frees 3 ms of RTT, which with the index used above is worth about 300 km more distance from the server, and may be cheaper to obtain than a new point of presence. That holds, however, only if the network leg is already under budget: if it is not, no local optimisation will recover it.

WebRTC as the transport

WebRTC carries, on the same stack, low-latency video from the server to the player and a data channel in the opposite direction. It includes path negotiation (ICE, with STUN and TURN servers to cross NATs), and it encrypts the traffic. For the video you choose the codecs and delegate congestion control to the browser; for the inputs you open a data channel. The delicate choice is the reliability of that channel.

The W3C specification defines the options ordered (true by default) and maxRetransmits for the data channel; RFC 8831 explains that underneath is SCTP with unordered delivery and partial reliability. For a movement command, retransmitting a lost packet delivers a stale input when newer ones have already arrived, so you use ordered: false and maxRetransmits: 0. The fragment below is the browser-side sketch: I compiled it with the TypeScript compiler and the DOM library, but I did not run it because it needs a browser and a signalling server.

// Sketch to adapt: needs a browser and a signalling server. Checked with the compiler only (DOM lib).
interface IceConfig { stunUrl: string; turnUrl: string; username: string; credential: string; }   // the TURN credentials come from your server, short-lived

function createSession(ice: IceConfig, video: HTMLVideoElement): { pc: RTCPeerConnection; input: RTCDataChannel } {
  const pc = new RTCPeerConnection({
    iceServers: [{ urls: ice.stunUrl }, { urls: ice.turnUrl, username: ice.username, credential: ice.credential }],
    bundlePolicy: 'max-bundle',
  });
  // Input: unordered and without retransmissions. A stale retransmitted command is worse than a lost one.
  const input = pc.createDataChannel('input', { ordered: false, maxRetransmits: 0 });
  pc.addTransceiver('video', { direction: 'recvonly' });
  pc.ontrack = (ev) => { video.srcObject = new MediaStream([ev.track]); };
  return { pc, input };
}

async function negotiate(pc: RTCPeerConnection, post: (offerSdp: string) => Promise<string>): Promise<void> {
  await pc.setLocalDescription(await pc.createOffer());
  const answer = await post(pc.localDescription!.sdp);
  await pc.setRemoteDescription({ type: 'answer', sdp: answer });
}

Two notes on that code. TURN credentials are not written in the client: they come from your server, short-lived, because a static credential in the JavaScript bundle is public. And the recvonly direction declares that this side receives the video and does not send it: the encoding part lives on the server.

Unordered channel: the sequence number

If packets can arrive out of order, the server cannot trust the order of arrival. Every input carries a sequence number, and the server applies only what is newer than the last one applied. The number is 16 bits and restarts from zero, so "newer" is decided on the circular distance: the listing uses ((a - b) & 0xffff) < 0x8000, the classic comparison for sequence numbers that wrap around. The packet is 11 bytes: two for the sequence, eight for the two floating-point movements and one for the button bitmask.

// Input packet: 2 bytes of sequence number + 2 float32 (movement) + 1 byte of buttons = 11 bytes
interface Input { seq: number; dx: number; dy: number; buttons: number; }
function encode(i: Input): ArrayBuffer {
  const buf = new ArrayBuffer(11), v = new DataView(buf);
  v.setUint16(0, i.seq & 0xffff); v.setFloat32(2, i.dx); v.setFloat32(6, i.dy); v.setUint8(10, i.buttons);
  return buf;
}
function decode(buf: ArrayBuffer): Input {
  const v = new DataView(buf);
  return { seq: v.getUint16(0), dx: v.getFloat32(2), dy: v.getFloat32(6), buttons: v.getUint8(10) };
}
// The sequence number is 16 bits and wraps around to zero: "a is newer than b" is decided on the circular distance
const isNewer = (a: number, b: number) => a !== b && ((a - b) & 0xffff) < 0x8000;

// Seeded generator (mulberry32)
function rng(seed: number) { return () => { seed |= 0; seed = (seed + 0x6D2B79F5) | 0; let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
  t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; }
const r = rng(5);

// 600 inputs (10 s at 60 Hz) on an unordered channel: variable delay, 2% lost
const sent = Array.from({ length: 600 }, (_, n) => ({ at: n * (1000 / 60), pkt: encode({ seq: 65_300 + n, dx: n, dy: 0, buttons: 0 }) }));
const arrivals = sent.filter(() => r() > 0.02).map(s => ({ at: s.at + 20 + r() * 40, pkt: s.pkt })).sort((a, b) => a.at - b.at);

let lastApplied = -1, backwardsNaive = 0, backwardsSeq = 0, dropped = 0, lastSeq = -1;
for (const a of arrivals) {
  const i = decode(a.pkt);
  // Strategy 1: apply everything in arrival order. Count how many times the command "goes backwards"
  if (lastApplied !== -1 && i.dx < lastApplied) backwardsNaive++;
  lastApplied = i.dx;
  // Strategy 2: apply only packets newer than the last one applied
  if (lastSeq !== -1 && !isNewer(i.seq, lastSeq)) { dropped++; continue; }
  lastSeq = i.seq;
}
console.log(`${arrivals.length} packets arrived out of 600; the sequence wraps through zero at 65536`);
console.log(`applied in arrival order: the state goes backwards ${backwardsNaive} times`);
console.log(`with a sequence number: ${dropped} old packets discarded, state never goes backwards`);
console.log(`packet: ${encode({ seq: 1, dx: 0, dy: 0, buttons: 0 }).byteLength} bytes`);

// Output:
// 584 packets arrived out of 600; the sequence wraps through zero at 65536
// applied in arrival order: the state goes backwards 96 times
// with a sequence number: 99 old packets discarded, state never goes backwards
// packet: 11 bytes

The simulation sends 600 inputs at 60 Hz with 20-60 ms of variable delay and 2% loss, with the sequence starting at 65,300 and therefore wrapping through zero halfway. Of the 600, 584 arrive. Applied in arrival order, the state goes backwards 96 times: the player would see the character jerk backwards. With the sequence number the server discards 99 packets older than the last one applied and the state never regresses. The two numbers (96 and 99) measure different things: the first counts the steps backwards relative to the previous packet, the second the packets older than the maximum already seen.

Encoding on the server

The frame drawn by the GPU has to be compressed before it leaves, and compression costs time. NVIDIA's hardware encoder exposes four tuning profiles: high quality, low latency, ultra-low latency and lossless. The Video Codec SDK guide associates cloud gaming with low latency at constant bitrate (CBR), and with ultra-low latency when the channel is tightly limited. I have no GPU in this machine, so I report no encoding times: you measure them, on your content and your card.

• Low-latency tuning with CBR: this is the setting the documentation indicates for cloud gaming
• No B-frames: a B-frame depends on a later one, so the encoder has to hold back the output and the guide requires extra buffers (1 + N_B) for this reason
• One negotiated codec per session: H.264 is the most widespread in hardware decoding on devices, AV1 compresses better but requires devices that can decode it

The three items lead to the same rule: every feature that improves compression by looking at the future costs latency. That is fine for streaming a film and not for a system in which the future is the player's next key press. The last item is a selection criterion, not a measurement: check on the devices you care about which hardware decoders are present.

Edge: choose the node with a probe, not with a map

Bringing the server closer to the player reduces the only stage that depends on geography. The naive way to choose a node is to translate the IP address into a location and take the closest one as the crow flies. It works in most cases, because distance sets the minimum, but the real path depends on how the operators are interconnected, not on geometry. A more robust choice is to have the client measure the RTT to two or three candidates with a short probe exchange and pick the best among those with free capacity.

Capacity is the other criterion: a nearby but saturated node gives a worse experience than one slightly farther away and free. You need to decide beforehand which occupancy threshold declares a node "full" and what happens to the next session (queue, secondary node, refusal with a clear message). I do not report a simulation of this choice: the numbers would depend entirely on the path stretch factors that I would have to invent. The right choice is tested on your own network.

Adaptive quality: hysteresis prevents flicker

The browser applies congestion control to the video on its own. Your application layer intervenes on something coarser: resolution and maximum bitrate (the W3C specification provides a maxBitrate in a sender's encoding parameters). A simple rule is to associate with each level an RTT threshold beyond which it is no longer suitable. The flaw is well known: if the RTT oscillates near a threshold, the system switches level continuously, and an image that jumps in quality is more annoying than a stable, lower quality.

interface Level { name: string; kbps: number; maxRtt: number; }   // a level is suitable as long as the RTT stays below maxRtt
const LEVELS: Level[] = [
  { name: 'high', kbps: 8000, maxRtt: 40 }, { name: 'medium', kbps: 4000, maxRtt: 70 }, { name: 'low', kbps: 2000, maxRtt: 120 },
];
const pick = (rtt: number) => LEVELS.findIndex(l => rtt < l.maxRtt) === -1 ? LEVELS.length - 1 : LEVELS.findIndex(l => rtt < l.maxRtt);

// Bare thresholds: switch as soon as the value crosses a threshold
function naive(trace: number[]): number {
  let level = 0, changes = 0;
  for (const rtt of trace) { const t = pick(rtt); if (t !== level) { level = t; changes++; } }
  return changes;
}
// With hysteresis: drop at once, but climb back only after 'hold' consecutive samples below the threshold with a 15% margin
function hysteresis(trace: number[], hold: number): number {
  let level = 0, changes = 0, good = 0;
  for (const rtt of trace) {
    const t = pick(rtt);
    if (t > level) { level = t; changes++; good = 0; }                        // getting worse: react at once
    else if (t < level && rtt < LEVELS[level - 1].maxRtt * 0.85) {            // getting better, with a margin
      if (++good >= hold) { level--; changes++; good = 0; }
    } else good = 0;
  }
  return changes;
}

function rng(seed: number) { return () => { seed |= 0; seed = (seed + 0x6D2B79F5) | 0; let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
  t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; }
const r = rng(3);
// 300 seconds, one sample per second: RTT around 40 ms with ±8 ms of noise (the "near the threshold" case)
const trace = Array.from({ length: 300 }, () => 40 + (r() - 0.5) * 16);
console.log('Mean RTT of the trace:', (trace.reduce((a, b) => a + b, 0) / trace.length).toFixed(1), 'ms');
console.log('level changes with bare thresholds:', naive(trace));
console.log('level changes with hysteresis (hold=5):', hysteresis(trace, 5));

// Output:
// Mean RTT of the trace: 40.2 ms
// level changes with bare thresholds: 145
// level changes with hysteresis (hold=5): 1

Over 300 RTT samples around 40 ms with 8 ms of noise, all right next to the threshold between "high" and "medium", bare thresholds switch level 145 times and the hysteresis rule only once. The hysteresis rule drops at once when things get worse and climbs back only after five consecutive samples below the threshold with a 15% margin. The number depends on the synthetic trace I generated, but the mechanism is general: react quickly on the way down and cautiously on the way up. In a real system it is also worth watching packet loss and the estimated bitrate, not only the RTT, because a low RTT does not guarantee enough bandwidth. How to collect these measurements without making the metrics explode is in the article on game backend observability.

Sharing the GPU

The GPU is the expensive resource, and one session per GPU is the simplest and most isolated choice. To increase density there are several roads: virtualisation with vGPU, hardware partitioning with MIG, or several processes on the same card. NVIDIA's MIG guide describes the partitioning of GPUs from the Ampere architecture onwards into up to seven separate instances, pointing to CUDA workloads; it says nothing about game rendering, so I do not present it as a solution for cloud gaming. The "sessions per GPU" and "overhead" figures depend on the game and the card, and I report none.

The number you need is yours and is measured: the cost of a session is the hourly cost of the GPU divided by the number of sessions that run together without exceeding the latency budget. Every extra session on the card lengthens rendering and encoding times for all of them, so you go up until the slowest stage is still inside the budget, and you stop there.

When cloud gaming is not the answer

If the players are spread where you cannot have a node a few milliseconds away, or if the game requires very quick reactions, the calculation in the first listing will tell you early that the budget does not add up. In that case a thin client with the game running locally and the server for shared state is better.

Frequently asked questions

What you can do now

You can set the latency budget of your game and compute how much distance your server can afford. You can configure the WebRTC input channel, and you know why a circular sequence number is needed. You know which encoder settings the documentation indicates for cloud gaming and which choices cost latency. You can avoid the flicker of adaptive quality with hysteresis. Still to be measured on your own system: encoding times, the real RTT of the paths and the number of sessions per GPU.