03 - OTP and GenServer: Distributed State Management with Integrated Fault Tolerance

A GenServer is a process that holds a state and answers messages, one at a time. Two consequences follow from this sentence that are worth more than all the theory: the state inside a GenServer needs no locks, because a single process touches it, and the GenServer is a bottleneck, because messages are served in a queue. Whoever designs well with OTP exploits the first and keeps an eye on the second.

This article writes a counter, then a rate limiter with structured state, then a per-user session registered by name, and in each case it says what the line that matters does and what happens if you remove it. At the centre is "let it crash", which is not an attitude but a division of labour between your code and the supervisor.

What OTP is, said without the name

OTP stands for Open Telecom Platform, and it is a name that misleads today: it has nothing to do with telephony and it is not a web framework. It is a set of behaviours - interfaces that your module implements - plus the rules on how to compose them into a tree of processes that watch over one another.

A behaviour solves a very concrete problem. Writing by hand a process that holds a state means writing a recursive loop with a receive inside, and in that loop you have to remember to handle system messages, synchronous requests with the right sender, orderly shutdown. GenServer is that loop, written once and battle-tested: you supply only the functions that decide what to do with each message.

A complete GenServer, line by line

The counter is the minimal example that contains everything: two synchronous operations, two asynchronous ones, an external message. The convention to note before the code is the split into two halves: the functions of the client API run in the process of whoever calls, the callbacks run in the process of the server. They are two different worlds living in the same file.

defmodule MyApp.Counter do
  use GenServer

  # --- Client API: runs in the CALLER's process ---

  def start_link(initial_value \\ 0) do
    GenServer.start_link(__MODULE__, initial_value, name: :my_counter)
  end

  def increment(n \\ 1), do: GenServer.call(:my_counter, {:increment, n})

  def get_value, do: GenServer.call(:my_counter, :get_value)

  def decrement_async(n \\ 1), do: GenServer.cast(:my_counter, {:decrement, n})

  def reset, do: GenServer.cast(:my_counter, :reset)

  # --- Callbacks: run in the SERVER's process ---

  @impl true
  def init(initial_value) do
    {:ok, initial_value}
  end

  @impl true
  def handle_call({:increment, n}, _from, state) do
    new_state = state + n
    {:reply, new_state, new_state}   # {:reply, what_i_answer, new_state}
  end

  def handle_call(:get_value, _from, state) do
    {:reply, state, state}           # I reply and leave the state as it was
  end

  @impl true
  def handle_cast({:decrement, n}, state) do
    {:noreply, state - n}            # no reply, only the new state
  end

  def handle_cast(:reset, _state), do: {:noreply, 0}

  @impl true
  def handle_info(:log_state, state) do
    IO.puts("[Counter] value: #{state}")
    {:noreply, state}
  end

  # Safety net: an unknown message must not bring the server down
  def handle_info(msg, state) do
    IO.puts("unexpected message: #{inspect(msg)}")
    {:noreply, state}
  end
end

The shape of the return values is the part that cannot be improvised. handle_call/3 returns {:reply, reply, new_state}: three elements, because it has to say both what to send to the caller and how the state stays. In {:increment, n} the two coincide - I reply with the new value - but it is not a rule: in :get_value I reply with the state and the state does not change. handle_cast/2 instead returns {:noreply, new_state}, two elements, because nobody is waiting.

The second parameter of handle_call, written here _from, is the pair that identifies the caller. The underscore says you do not use it, and in the majority of cases that is correct: it is needed when you want to reply later, returning {:noreply, state} and calling GenServer.reply(from, reply) afterwards. It is the way of not blocking the queue while you wait for something slow.

The second clause of handle_info/2, the one that accepts any message, is the only defensive line worth writing in a GenServer. Without it, an unforeseen message - the late reply to a request that timed out, a :DOWN from a monitor you were not expecting - matches no clause and the process dies with FunctionClauseError. With it, the same message is logged and the server carries on.

Beware instead of terminate/2, which was in the original of this article and has been deliberately removed here: it is invoked only if the process is trapping exits or if it shuts down in an orderly way. A Process.exit(pid, :kill), or a crash, skips it. Putting the writing of the state to disk in terminate/2 means losing that write exactly in the cases for which you put it there.

{:ok, _pid} = MyApp.Counter.start_link(0)

MyApp.Counter.increment()         # => 1   (call: waits and gets the value)
MyApp.Counter.increment(5)        # => 6
MyApp.Counter.decrement_async(2)  # => :ok (cast: returns at once, 2 not yet subtracted)
MyApp.Counter.get_value()         # => 4   (the call queues up AFTER the cast)

# A message sent by hand arrives in handle_info
send(:my_counter, :log_state)
# [Counter] value: 4

The line to look at is the third. decrement_async/1 returns :ok immediately, and at that moment the subtraction has not happened yet: you have only put a message in the server's mailbox. Yet the get_value() right after reads 4, that is the correct value. It is not luck: the GenServer serves the mailbox in order of arrival, so the cast is processed before the call that followed it.

This ordering guarantee holds between two messages from the same sender. If the cast started from another process, nobody assures you which of the two arrives first, and this is where tests that pass nine times out of ten come from.

The original version of this example put a :timer.sleep(10) between the two lines to "wait for the cast": I removed it because it teaches the wrong technique - a pause is not a synchronization, it is a bet that ten milliseconds are enough.

The correct way to wait for a cast, if you really must, is to follow it with any call to the same process: when the call replies, all the messages you sent before have been processed. It is exactly what the get_value() line above does, and that is why it works without pauses.

Structured state: a rate limiter

An integer as state is fine for a counter. As soon as the state has more than one field a struct is worth it, for two practical reasons: the fields have a name, and a typo on a name becomes a compile error instead of a new key added silently.

defmodule MyApp.RateLimiter do
  use GenServer

  defstruct window_ms: 60_000, max_requests: 100, requests: %{}

  def start_link(opts \\ []) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end

  @doc "Returns {:ok, remaining} or {:error, :rate_limited, retry_in_ms}"
  def check(user_id), do: GenServer.call(__MODULE__, {:check, user_id})

  @impl true
  def init(opts) do
    # Periodic cleanup: sends :cleanup to this very process every 30 s
    :timer.send_interval(30_000, :cleanup)

    {:ok, %__MODULE__{
      window_ms: Keyword.get(opts, :window_ms, 60_000),
      max_requests: Keyword.get(opts, :max_requests, 100)
    }}
  end

  @impl true
  def handle_call({:check, user_id}, _from, state) do
    now = System.monotonic_time(:millisecond)
    window_start = now - state.window_ms

    # I keep only the timestamps inside the window
    recenti =
      state.requests
      |> Map.get(user_id, [])
      |> Enum.filter(&(&1 > window_start))

    if length(recenti) >= state.max_requests do
      riprova_fra = List.last(recenti) + state.window_ms - now
      {:reply, {:error, :rate_limited, riprova_fra}, state}
    else
      nuovo_stato = put_in(state.requests[user_id], [now | recenti])
      {:reply, {:ok, state.max_requests - length(recenti) - 1}, nuovo_stato}
    end
  end

  @impl true
  def handle_info(:cleanup, state) do
    window_start = System.monotonic_time(:millisecond) - state.window_ms

    vivi =
      state.requests
      |> Enum.reject(fn {_user, ts} -> Enum.all?(ts, &(&1 <= window_start)) end)
      |> Map.new()

    {:noreply, %{state | requests: vivi}}
  end
end

The heart is System.monotonic_time/1, and the choice is not indifferent: the monotonic clock never goes backwards, while System.system_time/1 follows the system time and can jump when NTP corrects the machine. With the system time, a one-minute correction backwards would unblock all the users limited at that moment. To measure intervals you always use monotonic time.

The line Enum.filter(&(&1 > window_start)) is the sliding window: instead of zeroing a counter every minute, you keep the individual instants and count the recent ones. The difference shows at the boundary: with a counter zeroed at fixed intervals, a user can make 100 requests at the end of a minute and another 100 at the beginning of the next, that is 200 in two seconds. With the sliding window, no.

handle_info(:cleanup, ...) exists because handle_call filters the timestamps only of the users who come back. A user who makes ten requests and then disappears leaves their key in the map forever: without the cleanup, state.requests is a slow memory leak, proportional to the number of distinct users you have ever seen. :timer.send_interval/2 called inside init/1 is what triggers it, and it sends the message to the process that called it.

Two corrections compared with the previous version of this example, because they were wrong and there is no point in hiding it.

The retry_after was computed with hd(recenti), the head of the list, which is the most recent timestamp since insertion is at the head: it told the client to retry after a whole window instead of at the expiry of the oldest one, now it is List.last/1. And the remaining count was computed on a list to which the current instant had already been added, so it overestimated by one.

A limit remains that has to be stated, because it is architectural and cannot be fixed with one line: this rate limiter is a single process with the name __MODULE__, so every check of every user goes through that queue. As long as the callback costs microseconds it is fine; under real load you split by user, or move the counting into ETS, which can be read in parallel.

A comparison of the ways to handle a request limit in an HTTP service is in rate limiting on APIs.

"Let it crash": who handles what

The principle is misunderstood because the name sounds like giving up. It says instead one precise thing: handle the errors that are part of the domain, and let the process die for everything else, because a dead process is replaced by a healthy one and a process that carries on with an inconsistent state is not.

# Defensive: every call wrapped in a case, and a final rescue
def process_order_difensivo(order) do
  try do
    case validate_order(order) do
      {:ok, valido} ->
        case save_to_db(valido) do
          {:ok, salvato} ->
            case send_confirmation(salvato) do
              {:ok, _} -> {:ok, salvato}
              {:error, e} -> handle_email_error(e)
            end
          {:error, e} -> handle_db_error(e)
        end
      {:error, e} -> handle_validation_error(e)
    end
  rescue
    e -> handle_unexpected_error(e)
  end
end

# With `with`: the happy path is flat, and only the expected is handled
def process_order(order) do
  with {:ok, valido}  <- validate_order(order),
       {:ok, salvato} <- save_to_db(valido),
       {:ok, _}       <- send_confirmation(salvato) do
    {:ok, salvato}
  else
    {:error, :invalid_data} = errore ->
      Logger.warning("invalid order: #{inspect(order)}")
      errore                      # domain error: I return it

    {:error, :duplicate_order} ->
      Logger.info("duplicate order ignored: #{order.id}")
      {:ok, :duplicate}           # expected case: it is not an error
  end
end

with is the construct that makes the principle practicable. Every line with <- is a match: if it matches, the variable is bound and you move on to the next line; if it does not match, the whole with jumps to the else block with that value. The result is that the three operations in sequence read as a sequence, instead of nesting three levels deep as in the first version.

The substantial difference between the two blocks is not the indentation: it is that the first has a final rescue, and the second does not. That rescue catches everything, including the errors that signal a bug - a MatchError, a missing function clause - and turns them into a return value. The process survives with a state about which you know nothing any more, and the defect that caused it leaves no trace.

It is the trade-off that matters: the rescue buys immediate availability and pays in diagnosability. A process that survives a bug is a process that will repeat it, silently, until somebody notices the wrong data.

In the second block the else lists only two cases, and that is intended: they are the two that the domain foresees. If save_to_db returns {:error, :connection_refused}, no clause matches and with raises a WithClauseError: the process dies, the supervisor restarts it with a clean state, and in the meantime the line ends up in the logs. This is the delegation, and it works only if there really is a supervisor above - without a supervision tree, "let it crash" is just "crash".

Note the :duplicate_order case, which returns {:ok, :duplicate}: a duplicate order is not an error, it is a foreseen outcome of a network that retries. Distinguishing the two things is the hard part of the trade, more than the syntax. The topic has a name, idempotency, and a separate treatment in consumer idempotency.

The dividing line, in practice, falls like this:

• Invalid input data: handle, it comes from outside and it will happen
• Duplicate, resource not found, permission denied: handle, they are domain outcomes
• Database unreachable, external service down: let it propagate, it is not your process that can fix it
• MatchError, FunctionClauseError, ArithmeticError: let it propagate, they are bugs and must be seen
• "It should never happen": let it propagate, especially in this case

The single criterion under the five lines is this: handle what you can decide something sensible about, let the rest die. The first two have an answer to give to the caller; the last three do not, and pretending to have one means returning an invented value.

The most important line is the last, because it is the one you are most tempted to cover with a rescue: if a situation that should never happen happens, your model of reality is wrong, and the worst thing is to carry on as if nothing had happened.

Many instances of the same server: Registry

So far the servers had a fixed name, an atom, and therefore existed in a single copy. That is fine for a global rate limiter, not for something that exists per user. The name cannot be an atom - among other things atoms are not freed by the garbage collector, so generating them from user data is a slow way of exhausting memory - and you need a registry.

defmodule MyApp.UserSession do
  use GenServer

  def start_link(user_id) do
    GenServer.start_link(__MODULE__, %{user_id: user_id}, name: via(user_id))
  end

  # The key is any term: no atom created at runtime
  defp via(user_id), do: {:via, Registry, {MyApp.SessionRegistry, {:session, user_id}}}

  def get_session(user_id), do: GenServer.call(via(user_id), :get)
  def update(user_id, campi), do: GenServer.cast(via(user_id), {:update, campi})

  @impl true
  def init(state), do: {:ok, state}

  @impl true
  def handle_call(:get, _from, state), do: {:reply, state, state}

  @impl true
  def handle_cast({:update, campi}, state), do: {:noreply, Map.merge(state, campi)}
end

{:via, Registry, {registry_name, key}} replaces the atom as the name of the process. From that moment GenServer.call accepts that tuple in place of a PID, and the Registry acts as a phone book: it translates the key into the current PID. The gain is not only being able to have many instances, it is that the caller does not have to keep the PID - which changes at every restart of the process - but only the key, which does not change.

The private function via/1 exists so as not to repeat that tuple in four places. If you wrote it by hand in every function of the client API, one day you would change the name of the registry in three places out of four, and the fourth would look in a phone book that does not exist. The key here is the tuple {:session, user_id} and not an interpolated string: it avoids colliding with other families of processes that use the same registry.

defmodule MyApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      # keys: :unique -> a single instance per key
      {Registry, keys: :unique, name: MyApp.SessionRegistry},

      # The children are not known at startup: they are created when a user enters
      {DynamicSupervisor, name: MyApp.SessionSupervisor, strategy: :one_for_one}
    ]

    Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
  end
end

defmodule MyApp.SessionManager do
  def start_session(user_id) do
    DynamicSupervisor.start_child(MyApp.SessionSupervisor, {MyApp.UserSession, user_id})
  end

  def stop_session(user_id) do
    case Registry.lookup(MyApp.SessionRegistry, {:session, user_id}) do
      [{pid, _}] -> DynamicSupervisor.terminate_child(MyApp.SessionSupervisor, pid)
      [] -> {:error, :not_found}
    end
  end

  def active_sessions, do: DynamicSupervisor.which_children(MyApp.SessionSupervisor) |> length()
end

The order of the children is significant: the Registry comes before the DynamicSupervisor because a UserSession that starts must find the registry already up in order to register. Supervisor starts the children in the declared order and, with strategy: :one_for_one, restarts only the one that falls.

Swapping the two, a session starting in the first instant would find MyApp.SessionRegistry not yet started and would die on Registry.register.

DynamicSupervisor is the variant for children you do not know at startup. A normal Supervisor receives the list of children and starts them all; here the sessions are born when a user enters, so the list is empty at the beginning and grows with start_child/2. keys: :unique in the Registry is what guarantees a single session per user: with keys: :duplicate the same key would accept several processes and GenServer.call on the :via tuple would not know whom to talk to.

In stop_session/1, Registry.lookup/2 returns a list, not a single value: the empty list is the "session not active" case, and it is the one the case handles. Taking for granted that there is always an element - by writing [{pid, _}] = Registry.lookup(...) - would produce a MatchError every time somebody closes a session that is already closed, that is all the time.

Testing a GenServer without sleeping

A GenServer is a process, so its tests have one more problem than the tests of a pure function: the state survives from one test to the next. The solution is not to give it a fixed name in the tests and to start a new one for each.

defmodule MyApp.CounterTest do
  use ExUnit.Case, async: true

  setup do
    # No `name:`: the server is reachable only by PID,
    # so parallel tests do not disturb one another
    {:ok, pid} = GenServer.start_link(MyApp.Counter, 0)
    {:ok, counter: pid}
  end

  test "starts from zero", %{counter: pid} do
    assert GenServer.call(pid, :get_value) == 0
  end

  test "adds up the increments", %{counter: pid} do
    GenServer.call(pid, {:increment, 5})
    GenServer.call(pid, {:increment, 3})
    assert GenServer.call(pid, :get_value) == 8
  end

  test "the asynchronous reset is visible to the next call", %{counter: pid} do
    GenServer.call(pid, {:increment, 10})
    GenServer.cast(pid, :reset)
    # No sleep: the call queues up AFTER the cast of the same process
    assert GenServer.call(pid, :get_value) == 0
  end

  test "a hundred concurrent increments are not lost", %{counter: pid} do
    1..100
    |> Enum.map(fn _ -> Task.async(fn -> GenServer.call(pid, {:increment, 1}) end) end)
    |> Enum.each(&Task.await/1)

    assert GenServer.call(pid, :get_value) == 100
  end
end

The absence of name: in the setup is what makes async: true legitimate. With a registered name, two parallel tests would try to register the same atom and the second would fail with :already_started; without a name, every test has its own process and its own state. It is the technical reason why the name should be a parameter of start_link, not a constant written in the module.

The third test is the one worth reading twice. In the previous version it had a :timer.sleep(10) between the cast and the assert, and it passed; here the sleep is not there and it passes anyway, because the final call queues up behind the cast sent from the same test process.

A test with a sleep inside is slow when it works and intermittent when the machine is loaded: synchronization must be obtained from the protocol, not from the stopwatch.

The last test demonstrates the property we started from. A hundred processes call {:increment, 1} together and the total is exactly 100, without a line of synchronization code: the GenServer serves one request at a time. In a language with shared memory the same test would require a lock or an atomic integer, and written badly it would lose increments.

What you can do now

You can choose between call and cast by looking at whether the caller needs the reply, hold a structured state with a struct, give birth to a process per key with Registry and DynamicSupervisor, and tell the errors to handle from those to let propagate. You also know two things that cost dearly to whoever does not: the call expires after five seconds, and terminate/2 is not guaranteed.

The fundamentals of the language on which all this rests - pattern matching, pipe, immutability - are in Elixir from scratch; the model of processes and messages that the GenServer encapsulates is in BEAM and processes. On the limit that remains open, that is the single queue in front of a process, the general problem and its vocabulary are treated in backpressure and flow control.

Sources

• Elixir - GenServer behaviour - signatures and allowed return values of the callbacks, the default 5000 ms timeout of call/2, the conditions in which terminate/2 is not invoked, and GenServer.reply/2 for the deferred reply. Consulted on 2026-11-08.
• Elixir - Registry - the semantics of keys: :unique versus :duplicate, the shape of the tuple {:via, Registry, {registry, key}} and the fact that lookup/2 returns a list. Consulted on 2026-11-08.
• Elixir - DynamicSupervisor - difference from Supervisor for children started at runtime, and start_child/2. Consulted on 2026-11-08.
• Erlang/OTP - Design Principles - the definition of behaviour, supervision trees and restart strategies, that is the context that makes "let it crash" make sense. Consulted on 2026-11-08.
• Joe Armstrong, "Making reliable distributed systems in the presence of software errors" (2003) - the origin of the separation between business code and recovery code, from which the principle descends. Consulted on 2026-11-08.

The first three are the documentation of the modules used in the code, and they are the source to consult for any doubt about a signature, because they follow the version of the language. The last two serve a different question: not "how is it written" but "why is it written this way". The fourth is the one to read if one day you find yourself having to defend the choice of not intercepting an error, because it is the document that founds it.

A clarification on the numbers of this article: the only quantitative value that appears is the 5000 ms timeout, and it comes from the documentation cited above. The claim that the three callbacks "cover 95% of use cases", present in the previous version, has been removed: there is no measurement behind it, and nobody has counted those cases.