79 - Gymnasium in Practice: Taxi, FrozenLake and the Agent That Memorises the Simulator

So far we have written the environments ourselves: a grid, a corridor, a cliff. In real work you use standard environments, supplied by a library, which let you compare different algorithms on the same problem. The reference library is Gymnasium, the maintained continuation of OpenAI Gym. Here you use Q-Learning on two of its classic environments, Taxi and FrozenLake, learning the API, an evaluation protocol that does not deceive, and a result to remember every time you train an agent in simulation: an agent can learn the simulator perfectly well without learning the task. The phenomenon can be measured: an agent that is 100% perfect on the training lake drops to 5.3% as soon as the lake becomes slippery and to 17.5% on other maps.

The Gymnasium vocabulary

Gymnasium defines a common interface for RL environments and contains many of them. The numbers in this article were obtained with version 1.4.0, installed in a temporary folder and not in the system environment; environment identifiers change between versions: with 1.4.0 Taxi-v3 is deprecated and you use Taxi-v4, so always check the name in your own installation. The observation and action spaces (observation_space, action_space) describe what the agent sees and what it can do: Discrete(n) is an integer from 0 to n − 1. A wrapper is a layer that modifies an environment without changing its code, like the TimeLimit that truncates long episodes, and the seed makes the environment's randomness reproducible: env.reset(seed=…).

An episode ends in two different ways. With terminated a terminal state of the problem has been reached (passenger delivered, fell into a hole); with truncated the episode was interrupted from outside (time limit) even though the state was not final. The Gymnasium usage guide in fact comments truncated as "true if the time limit is reached". Telling them apart matters for bootstrapping, as you will see in the code. The evaluation protocol is the procedure with which a policy is measured: fixed policy (ε = 0), new episodes with seeds never used in training, many episodes and a success metric defined beforehand. Finally overfitting to the simulator is the difference between performance in the training environment and in a slightly different one, the "real world"; in English this is called the sim-to-real gap.

The analogy: the student who memorises the answers

A student prepares for the exam by going over the same sheet of questions from last year's professor a thousand times, memorising the answers. If the exam questions are the same, he scores 30; if they are slightly different, he collapses. He has not learned the subject, he has learned that sheet. An agent always trained on the same simulator is in the same situation: the policy can be perfect there and fragile as soon as something changes (a slippery lake, another maze). With tabular Q-Learning it is even more pronounced: the "state" is just a cell number, so the table is the memorised map.

The Gymnasium API in five lines

The listing creates the Taxi, resets it with a seed, takes a random action and prints what the environment returns. The random action, env.action_space.sample(), has its own seed (env.action_space.seed(0)): the action space has a generator of its own, separate from the one used by reset, and without a seed the output would change on every run. With seed 0 the sampled action is 5 (drop off), illegal because the passenger is not in the taxi: the reward is −10 and the observation stays 314.

import numpy as np
import gymnasium as gym

env = gym.make("Taxi-v4")
env.action_space.seed(0)                           # the action space has its own generator
obs, info = env.reset(seed=0)
print("observation space:", env.observation_space, "| action space:", env.action_space)
print("first observation:", obs, "| info:", info)
obs, reward, terminated, truncated, info = env.step(env.action_space.sample())
print("after one step ->", dict(obs=obs, reward=reward, terminated=terminated, truncated=truncated))
print("episode time limit:", env.spec.max_episode_steps)
print("decoded state 0..:", list(env.unwrapped.decode(obs)), "(taxi row, taxi col, passenger location, destination)")

# Output (two consecutive runs, same output; gymnasium 1.4.0, numpy 2.5.4 on Python 3.14):
# observation space: Discrete(500) | action space: Discrete(6)
# first observation: 314 | info: {'prob': 1.0, 'action_mask': array([1, 1, 0, 0, 0, 0], dtype=int8)}
# after one step -> {'obs': 314, 'reward': -10, 'terminated': False, 'truncated': False}
# episode time limit: 200
# decoded state 0..: [3, 0, 3, 2] (taxi row, taxi col, passenger location, destination)

gym.make("Taxi-v4") creates the environment. The observation is an integer between 0 and 499, and the action an integer between 0 and 5: six actions (south, north, east, west, pick up, drop off). The 500 states are 25 taxi positions × 5 passenger positions (four stops plus "inside the taxi") × 4 destinations; the decode method breaks them apart. env.reset(seed=0) returns the first observation and an info dictionary (here with a mask of the possible actions). env.step(a) returns five values: observation, reward, terminated, truncated, info. The 200-step limit is a TimeLimit wrapper. The Taxi rewards, according to the documentation: −1 for every step, +20 for a successful delivery, −10 for illegal pick-up or drop-off.

Q-Learning with Gymnasium, and a detail that matters: terminated versus truncated

We rewrite the Q-Learning of the previous article on the Gymnasium API, with an exploration rate ε that decreases linearly from 1 to 0.05 (at the start we explore a lot, then exploit more), and an evaluate function that applies the correct protocol. At the end there is a reference: the random policy.

def q_learning_gym(env_id, episodes, alpha=0.1, gamma=0.99, eps_start=1.0, eps_end=0.05, eps_decay_episodes=None, seed=0, **make_kwargs):
    """Tabular Q-learning on any Gymnasium environment with Discrete observations and actions."""
    env = gym.make(env_id, **make_kwargs)
    n_s, n_a = env.observation_space.n, env.action_space.n
    rng = np.random.default_rng(seed)
    Q = np.zeros((n_s, n_a))
    eps_decay_episodes = eps_decay_episodes or episodes // 2
    returns = []
    for ep in range(episodes):
        epsilon = max(eps_end, eps_start - (eps_start - eps_end) * ep / eps_decay_episodes)    # linear decay
        s, _ = env.reset(seed=int(rng.integers(1 << 30)))
        total, done = 0.0, False
        while not done:
            a = int(rng.integers(n_a)) if rng.random() < epsilon else int(rng.choice(np.flatnonzero(Q[s] == Q[s].max())))
            s2, r, terminated, truncated, _ = env.step(a)
            # IMPORTANT: only `terminated` cuts the bootstrap. A time-limit `truncated` does not mean the state is worthless.
            target = r + (0.0 if terminated else gamma * Q[s2].max())
            Q[s, a] += alpha * (target - Q[s, a])
            s, total, done = s2, total + r, (terminated or truncated)
        returns.append(total)
    return Q, np.array(returns)

def evaluate(env_id, Q, episodes=200, seed=1000, **make_kwargs):
    """Greedy policy (epsilon = 0) on FRESH episodes: mean return and fraction of episodes that ended with a positive reward."""
    env = gym.make(env_id, **make_kwargs)
    totals, successes = [], 0
    for i in range(episodes):
        s, _ = env.reset(seed=seed + i)
        total, done = 0.0, False
        while not done:
            s, r, terminated, truncated, _ = env.step(int(np.argmax(Q[s])))
            total += r; done = terminated or truncated
        totals.append(total); successes += int(terminated and r > 0)     # `terminated` alone also covers falling into a hole!
    return float(np.mean(totals)), successes / episodes

# the random policy, as a reference
rng = np.random.default_rng(0)
env = gym.make("Taxi-v4")
env.action_space.seed(0)                           # make the random baseline reproducible
rand = []
for i in range(200):
    env.reset(seed=500 + i); total, done = 0.0, False
    while not done:
        _, r, terminated, truncated, _ = env.step(env.action_space.sample()); total += r; done = terminated or truncated
    rand.append(total)
print(f"random policy on Taxi: mean return {np.mean(rand):7.1f}")

for episodes in (500, 5000, 20000):
    Q, train_returns = q_learning_gym("Taxi-v4", episodes)
    mean_ret, success = evaluate("Taxi-v4", Q)
    print(f"{episodes:6d} training episodes: greedy evaluation mean return {mean_ret:6.2f}, deliveries completed {100 * success:5.1f}%")

# Output (two consecutive runs, same output; gymnasium 1.4.0, numpy 2.5.4 on Python 3.14):
# random policy on Taxi: mean return  -764.0
#    500 training episodes: greedy evaluation mean return -185.13, deliveries completed   7.0%
#   5000 training episodes: greedy evaluation mean return   7.96, deliveries completed 100.0%
#  20000 training episodes: greedy evaluation mean return   7.96, deliveries completed 100.0%

In q_learning_gym the loop is the one from the previous article; there are three novelties. The decay of ε: epsilon = max(eps_end, eps_start − (eps_start − eps_end)·ep/eps_decay_episodes), linear. Then the bootstrap that is cut only by terminated: target = r + (0 if terminated else γ·max Q[s2]). If the episode was truncated by the time limit, the next state is not "without a future": the problem would have continued, and zeroing the bootstrap in that case would teach the agent that states near the time limit are worth less than they are. It is a common and silent bug. The loop, however, stops on terminated or truncated. Finally each new episode starts with a seed drawn from the generator, so the training episodes are varied but reproducible.

evaluate is the protocol: ε = 0 (argmax), new episodes with seeds starting at 1000 (never used in training), 200 episodes. It defines success as "episode terminated with a positive reward" (terminated and r > 0): pay attention to this line, because in FrozenLake terminated is true also when you fall into a hole, and counting only terminated would give a 100% success to an agent that always falls. On the Taxi the random policy obtains an average of −764 (episodes that all last the full 200 steps, with many illegal −10 actions). After 500 training episodes the agent is still poor (−185, delivers 7% of the time). After 5000 it reaches +7.96 with 100% deliveries, and 20000 change nothing: it has converged. To know whether 7.96 is the maximum, we compute the exact optimal policy with value iteration on the model that Gymnasium exposes for "toy-text" environments (env.unwrapped.P, the same table as in the article on the MDP) and evaluate it with the same protocol.

def optimal_policy_from_model(env_id, gamma=1.0, **make_kwargs):
    """Gymnasium's toy-text environments expose their model in env.unwrapped.P, so we can compute the exact optimum for reference."""
    P = gym.make(env_id, **make_kwargs).unwrapped.P
    n_s, n_a = len(P), len(P[0])
    V = np.zeros(n_s)
    for _ in range(2000):
        for s in range(n_s):
            V[s] = max(sum(p * (r + gamma * V[s2] * (not t)) for p, s2, r, t in P[s][a]) for a in range(n_a))
    return np.array([[sum(p * (r + gamma * V[s2] * (not t)) for p, s2, r, t in P[s][a]) for a in range(n_a)] for s in range(n_s)])

# Is 7.96 the best possible on Taxi?  Compute the exact optimum from the model Gymnasium exposes, and evaluate it the same way.
Q_taxi_opt = optimal_policy_from_model("Taxi-v4", gamma=0.99)
print("exact optimal policy on Taxi: mean return %.2f, deliveries %.0f%%" % (evaluate("Taxi-v4", Q_taxi_opt)[0], 100 * evaluate("Taxi-v4", Q_taxi_opt)[1]))

# Output (two consecutive runs, same output; gymnasium 1.4.0, numpy 2.5.4 on Python 3.14):
# exact optimal policy on Taxi: mean return 7.96, deliveries 100%

The optimal_policy_from_model function runs value iteration (2000 sweeps, more than enough for these environments) using env.unwrapped.P and returns the q* table; the greedy policy with respect to it, evaluated with evaluate, obtains 7.96. The optimal value is therefore 7.96: the agent has learned the optimal policy, without ever seeing the model. It is a good health test: when possible, comparing with an exact reference tells you whether a result is "good" or merely "better than before".

FrozenLake: when even the optimal policy does not always win

FrozenLake is a 4×4 frozen lake: you start at S and must reach G without falling into the holes H. With is_slippery=True (the default) the ice is slippery: according to the documentation the move succeeds with probability 1/3 and deviates in each of the two perpendicular directions with probability 1/3. The reward is 1 only on reaching the goal, 0 otherwise: sparse and delayed reward, the case of the article on delayed rewards and reward hacking. Since the world is stochastic, even the best possible policy does not always reach the goal: you must know how often it does, otherwise you cannot judge Q-Learning.

# FrozenLake 4x4: walk from S to G over a frozen lake without falling into holes; on a slippery lake the move often slips sideways.

Q_opt = optimal_policy_from_model("FrozenLake-v1", map_name="4x4", is_slippery=True)
ok = evaluate("FrozenLake-v1", Q_opt, episodes=5000, map_name="4x4", is_slippery=True)
print(f"exact optimal policy (computed with the model): success rate {100 * ok[1]:.1f}%")

for episodes in (1000, 20000, 100000):
    Q, _ = q_learning_gym("FrozenLake-v1", episodes, alpha=0.05, gamma=0.99, map_name="4x4", is_slippery=True)
    m, success = evaluate("FrozenLake-v1", Q, episodes=5000, map_name="4x4", is_slippery=True)
    print(f"Q-learning, {episodes:6d} episodes: success rate {100 * success:5.1f}%")

# Output (two consecutive runs, same output; gymnasium 1.4.0, numpy 2.5.4 on Python 3.14):
# exact optimal policy (computed with the model): success rate 72.9%
# Q-learning,   1000 episodes: success rate   2.9%
# Q-learning,  20000 episodes: success rate  72.9%
# Q-learning, 100000 episodes: success rate  72.9%

The optimal greedy policy obtains 72.9% successes over 5000 episodes: it is the maximum for this slippery lake. Q-Learning, with α = 0.05 and decaying ε, is at 2.9% after 1000 episodes (it has almost never seen the goal), reaches 72.9% after 20000, exactly the optimum, and 100000 change nothing. Two lessons follow. Comparing with 72.9% and not with 100% is what gives the number its meaning. And the move from 2.9% to 72.9% shows the exploration problem with sparse reward: in the first episodes the agent never finds the prize and has nothing to learn from.

The agent that memorises the simulator

Now the part that matters. We will run two experiments on the same idea: train in one environment and evaluate in a slightly different one.

from gymnasium.envs.toy_text.frozen_lake import generate_random_map

# (1) train in a DETERMINISTIC simulator, deploy in a SLIPPERY one
Q_det, _ = q_learning_gym("FrozenLake-v1", 20000, alpha=0.5, gamma=0.99, map_name="4x4", is_slippery=False)
print("trained on the non-slippery lake:")
print("   success on the non-slippery lake:", f"{100 * evaluate('FrozenLake-v1', Q_det, 2000, map_name='4x4', is_slippery=False)[1]:5.1f}%")
print("   success on the SLIPPERY lake    :", f"{100 * evaluate('FrozenLake-v1', Q_det, 2000, map_name='4x4', is_slippery=True)[1]:5.1f}%   (the optimum there is 72.9%)")

# (2) train on ONE map, deploy on other maps of the same size (non-slippery, so that a perfect policy would score 100%)
train_map = generate_random_map(size=6, p=0.85, seed=0)
Q_map, _ = q_learning_gym("FrozenLake-v1", 20000, alpha=0.5, gamma=0.99, desc=train_map, is_slippery=False)
print("trained on one 6x6 map:")
print("   success on the map it was trained on :", f"{100 * evaluate('FrozenLake-v1', Q_map, 200, desc=train_map, is_slippery=False)[1]:5.1f}%")
others = []
for k in range(1, 41):
    other = generate_random_map(size=6, p=0.85, seed=k)
    others.append(evaluate("FrozenLake-v1", Q_map, 1, desc=other, is_slippery=False)[1])
print(f"   success on 40 OTHER random 6x6 maps   : {100 * np.mean(others):5.1f}%")

# Output (two consecutive runs, same output; gymnasium 1.4.0, numpy 2.5.4 on Python 3.14):
# trained on the non-slippery lake:
#    success on the non-slippery lake: 100.0%
#    success on the SLIPPERY lake    :   5.3%   (the optimum there is 72.9%)
# trained on one 6x6 map:
#    success on the map it was trained on : 100.0%
#    success on 40 OTHER random 6x6 maps   :  17.5%

In experiment 1 we train in a deterministic simulator (is_slippery=False) and then evaluate both there and on the slippery lake. In the training world the success is 100%; on the slippery lake 5.3%, against the 72.9% that an agent trained on the slippery lake can achieve. The agent has learned a sequence of precise moves, which works only if every move goes exactly where it should. In experiment 2 we train on a randomly generated 6×6 map (generate_random_map(size=6, p=0.85, seed=0)) and evaluate on 40 other maps of the same size (seeds 1 to 40, each once, in non-slippery mode, so that a perfect policy would obtain 100%). On the training map the success is 100%; on the others 17.5%.

Why? For the tabular agent "state" means "cell number": it knows nothing about the map, it has only learned "from cell 7 go right, from 8 go down…". On another map cell 7 has different holes around it, and since it does not see the map the agent cannot adapt: the layout of the holes does not exist in its state space. The 17.5% is not a coincidence: on certain maps the memorised path happens to be free of holes. I did not measure what a random policy would obtain on those maps, so I cannot say how far above chance it is. This is overfitting to the simulator, and it is not a defect of Q-Learning: it is a property of always training on the same environment.

The remedies used in practice each have their own cost. Domain randomisation trains on many variants of the environment (different maps, different friction, noise) so that the policy cannot memorise one of them, but it requires an agent able to see the difference. That is why what changes is put into the observation: if the map changes, the map must be part of the state, which is impossible with a table (the states explode) and feasible with a neural network that receives the map as input, the road of Deep Q-Learning. You then set aside maps, seeds and physical parameters for evaluation, as you set aside the test set: an agent is not "finished" until it has been tried outside its simulator. And if the policy is sensitive to small perturbations, it is not ready for the real world, whatever safety margin you estimate.

An evaluation protocol that does not deceive

1. Frozen policy (ε = 0, no updates during evaluation).
2. New episodes: seeds and (if possible) environment variants never used in training.
3. Many episodes (hundreds or thousands): in stochastic environments few episodes give noisy numbers.
4. Several independent trainings (different seeds of the agent), with mean and standard deviation: a single training can be lucky.
5. A success metric defined beforehand and distinct from the training reward (like "delivery completed", not "total reward").
6. A reference: the exact optimal policy, when it can be computed; otherwise a simple policy and a random one.

The six points read as a single defence against self-deception: the first three avoid measuring memory instead of ability, the fourth avoids mistaking a lucky training for a method, the last two give a yardstick for judging the number. In our experiments points 1, 2, 3 and 6 are respected; point 4 is not, because every training was done with a single agent seed. This is a declared limit of the reported numbers: on the Taxi and on the slippery FrozenLake the results coincide with the optimum, which makes it unlikely that they are due to luck, but does not prove it.

Trade-offs and pitfalls

The table is precise but fragile: on small, fixed problems it gives the optimum (7.96 on the Taxi, 72.9% on FrozenLake) and it does not generalise to what it has not seen. A single training environment is simple and fast, many make the policy robust but the learning harder. A decay of ε that is too fast stops the agent on a sub-optimal policy, too slow wastes episodes.

The pitfalls follow from here. Counting terminated as success makes an agent that always falls into the holes look "perfect". Zeroing the bootstrap on truncated teaches wrong values near the time limit. Evaluating with ε > 0 or with the same seeds as in training measures memory, not ability. Comparing with 100% when the maximum is 72.9% makes an optimal agent look poor. Thinking that a policy perfect in simulation is ready for the real world is the mistake of the 100% that becomes 5.3% with a little slipperiness. And forgetting that environment ids change between versions (Taxi-v3 against Taxi-v4) makes results non-reproducible: record the library version together with the numbers.

A worked example by hand: the Taxi encoding and how many episodes you need to trust a percentage

The state encoding. In the Taxi the state is a single integer that packs four pieces of information: state = ((taxi_row · 5 + taxi_column) · 5 + passenger_position) · 4 + destination. The observation 314 we saw is decoded like this: 314 mod 4 = 2 (destination 2), 314 div 4 = 78; 78 mod 5 = 3 (passenger in position 3), 78 div 5 = 15; 15 mod 5 = 0 (column 0), 15 div 5 = 3 (row 3). That is [3, 0, 3, 2], as printed by decode. The maximum state is ((4·5 + 4)·5 + 4)·4 + 3 = 499: hence the 500 states.

How many evaluation episodes are needed? A success percentage measured over n episodes is an estimate, with standard error √(p(1 − p)/n). For the optimal FrozenLake (p = 0.729): with n = 200 the standard error is √(0.729·0.271/200) = 0.031, that is ±6.2 percentage points at 95% (two standard errors); with n = 5000 it is 0.0063, that is ±1.2 points. That is why we evaluated FrozenLake over 5000 episodes and not 200: with 200, the difference between 70% and 75% would be within the noise. For the Taxi, where the policy is deterministic and the success is 100%, a few episodes are enough.

And the 17.5% on the "other maps"? They are 40 maps evaluated with a single episode each (non-slippery, hence deterministic): 17.5% = 7 successes out of 40, with standard error √(0.175·0.825/40) = 0.060, that is ±12 points. The result "far below 100%" is solid; the exact figure is approximate.

Exercises with solutions

1. Decode state 0 and state 499 of the Taxi.
   Solution: state 0: [0, 0, 0, 0]. State 499: [4, 4, 4, 3] (row 4, column 4, passenger 4, destination 3).
2. An agent has 90% success over 1000 evaluation episodes. What is the standard error?
   Solution: √(0.9·0.1/1000) = 0.0095, that is about ±1.9 points at 95%.
3. Why in FrozenLake do we not count terminated as success?
   Solution: because terminated is true also when falling into a hole: it must be counted as success only if the final reward is positive.
4. Should the bootstrap be zeroed for truncated?
   Solution: no: truncated indicates an external interruption (time), the state is not without a future. It is zeroed only for terminated.
5. The agent obtains 72.9% on the slippery FrozenLake: is that a good result?
   Solution: yes, 72.9% is the maximum attainable (optimal policy computed from the model); the judgement comes from the comparison with the optimum, not with 100%.

The first two exercises are calculations to do with pen and paper, and are to be rechecked with one line of Python (the solutions were recomputed that way). The other three are the points at which an RL experiment is falsified without any error appearing: if you can explain them, you can already read the results of the following articles critically.

Frequently asked questions

Why env.reset(seed=…) only on the first call?

The seed initialises the environment's generator; calling it at every episode with the same number would always produce the same episode. You pass reset a different seed per episode (as in evaluate) or set it only once.

What is the info field?

A dictionary with extra information not used by the agent (in the Taxi, the probability of the transition and a mask of valid actions). It must not be used to decide during evaluation unless it is part of the state accessible to the agent.

Can I use Gymnasium for a problem of my own?

Yes: you write a class that inherits from gym.Env and implements reset and step with defined observation and action spaces, and this is what lets the same algorithms work on different problems. I did not try this path here.

Why does the Taxi Q table succeed with 5000 episodes and not with 500?

Because there are 500 states × 6 actions = 3000 cells to learn and, with ε-greedy, each cell must be visited several times. With 500 episodes many have never been touched.

Is it normal that the return is 7.96 and not 20?

Yes: the delivery gives +20 but every previous step costs −1, so an average return of 7.96 corresponds to about 13 steps per episode (21 − 7.96 ≈ 13, because the last step is worth +20 in place of −1).

What you can do now

You can use the Gymnasium API (reset and step with five returned values), tell terminated from truncated and zero the bootstrap only for the former. You have seen tabular Q-Learning find the optimal policy in both environments: Taxi +7.96 (against −764 for the random policy) and FrozenLake 72.9%, equal to the maximum computed from the model. And you have seen the same agent, 100% perfect in its simulator, drop to 5.3% on a slippery lake and to 17.5% on other maps: it memorised the environment, not the task. That is why you evaluate with a frozen policy, new episodes and variants set aside, and compare with a reference. The table does not generalise and does not scale: the next article replaces the table with a neural network (Deep Q-Learning) and meets the new problems that this brings.