14 - The Future of AI Agents: Emergent Capabilities and Limitations

The problem: deciding on forecasts, with measurements in hand

Anyone deciding today whether to build an agent, hire to build one or entrust it with a process receives two kinds of messages: promises (autonomy on the way, growing markets, jobs that disappear) and failures told afterwards. Neither tells you what to do on Monday. This article closes the series by choosing another road: it starts from what can be measured or read in a primary source, says where forecasts are truly uncertain, and draws from it criteria for deciding. The market figures, project failure rates and adoption estimates that circulate are not here: I did not reach a primary source that supports them, and I preferred to leave them out.

What remains is four ideas, each with a mechanism and a proof. An agent's errors compound along the steps. "Emergent capabilities" also depend on how they are measured. Correction works when the signal comes from outside. And studies on work do not say the same thing as one another, which is already information. Three listings make them concrete; they use invented numbers, declared as such, and run with the Python 3.14.4 standard library alone, twice with identical output, without models.

Errors compound: the mathematics of steps

An agent does not answer: it executes a chain of steps, and the task succeeds only if the chain holds. If each step succeeds with probability p and the steps are independent, the task succeeds with probability p raised to the number of steps. It is a simplifying assumption (real steps are correlated, and some errors correct themselves), but it gives the order of magnitude of the problem, and it is what the article on testing measures with repeated attempts. The listing computes the table and a second case, in which a check discovers 80% of the errors and the step is repeated once (example values).

# Probability that ALL the steps of a task succeed, if each step succeeds with probability p
# and the steps are independent (a simplifying assumption, almost never true in real systems).
def all_succeed(p, n):
    return p ** n


def with_check(p, n, d):
    """After a wrong step, a check detects it with probability d and the step is repeated ONCE."""
    p_step = p + (1 - p) * d * p
    return p_step ** n


print(f"{'p per step':>11} | {'5 steps':>8} {'10 steps':>9} {'20 steps':>9}")
for p in (0.99, 0.95, 0.90):
    print(f"{p:>11.2f} | " + " ".join(f"{all_succeed(p, n):>8.3f}" for n in (5, 10, 20)))

print("\nwith a check that detects 80% of the errors (d = 0.8) and one repetition")
for p in (0.95, 0.90):
    print(f"{p:>11.2f} | " + " ".join(f"{with_check(p, n, 0.8):>8.3f}" for n in (5, 10, 20)))

# Output:
#  p per step |  5 steps  10 steps  20 steps
#        0.99 |    0.951    0.904    0.818
#        0.95 |    0.774    0.599    0.358
#        0.90 |    0.590    0.349    0.122
#
# with a check that detects 80% of the errors (d = 0.8) and one repetition
#        0.95 |    0.941    0.886    0.785
#        0.90 |    0.868    0.753    0.567

With 95% success per step, a ten-step task succeeds 6 times in 10 and a twenty-step one a little more than 1 in 3. That 95% looks like an excellent model and produces an agent that often fails. The second block shows the remedy the series has repeated: a check that discovers errors and a new attempt take the ten-step task from 60% to 89%. But they work only if the check is independent of the step it checks and if the step can be repeated without side effects (see the article on tool calling). It also applies to multi-agent architectures: the article on orchestration reports a study in which independent agents amplify errors far more than a structure with coordination.

Emergent capabilities: what it means and what it can hide

Wei and colleagues (2022) call a capability emergent if it is not present in smaller models but is present in larger ones, and which therefore cannot be predicted by extrapolating the performance of the smaller ones. The idea has guided many expectations about the future of agents: planning, correction, tool use appear at a certain scale. Schaeffer and colleagues (2023) propose an alternative explanation: when the same outputs are analyzed, the "sudden appearance" depends on the researcher's choice of metric, because nonlinear or discontinuous metrics produce apparent emergence, while continuous ones give smooth and predictable changes. The authors show this with a mathematical model and with tests on families of real models.

The mechanism is understood with a tabletop example. Imagine a family of models whose per-token accuracy grows regularly and a question whose answer is ten tokens long, valid only if all of them are right. The listing uses invented numbers, not those of the paper, to reproduce the argument: the left column is the continuous measure, the right one the exact match.

# Per-token accuracy of a family of models: INVENTED numbers, they grow smoothly with scale.
SCALE = [1, 2, 3, 4, 5, 6, 7, 8]
P_TOKEN = [0.60, 0.70, 0.78, 0.85, 0.90, 0.94, 0.97, 0.99]
ANSWER_LENGTH = 10   # the right answer is a sequence of 10 tokens

print(f"{'scale':>5} {'continuous metric':>17} {'exact match':>22}")
for s, p in zip(SCALE, P_TOKEN):
    exact = p ** ANSWER_LENGTH   # the answer counts only if ALL the tokens are right
    print(f"{s:>5} {p:>17.2f} {exact:>22.3f}")

# Output:
# scale continuous metric            exact match
#     1              0.60                  0.006
#     2              0.70                  0.028
#     3              0.78                  0.083
#     4              0.85                  0.197
#     5              0.90                  0.349
#     6              0.94                  0.539
#     7              0.97                  0.737
#     8              0.99                  0.904

The continuous metric rises from 0.60 to 0.99 without any jump. The exact match stays under 10% up to scale 3 and then rises steeply, to 0.90 at the eighth: looked at alone it seems a capability that "switches on". It is the same thing observed with another ruler. This does not prove that no capability is truly emergent (the authors limit their thesis to the cases they analyzed), but it teaches a practical rule to whoever builds: do not plan on a capability that "will arrive with the next model". Measure your task with a continuous metric (how many steps succeed, not only whether everything succeeds) and watch the trend on the models you can actually try.

Self-correction: it works with an external signal

The Reflexion pattern (Shinn and colleagues, 2023) has the agent observe the result of an attempt, write a reflection in words and keep it in a memory for the next attempt. In the abstract the authors report 91% pass@1 on HumanEval against the 80% of the then state-of-the-art GPT-4, and specify that the feedback signal can be external or internally simulated. Another work (Huang and colleagues, ICLR 2024) examines intrinsic correction, the kind done by the model without any external feedback, and concludes that in reasoning models struggle to correct themselves and sometimes get worse after the correction.

The two things do not contradict each other: the difference lies in where the signal comes from. The listing compares two loops on the same task, a function that must sum the even numbers of a list. The "model" is a script that proposes first a wrong version and then a right one; its self-critique always answers "ok", my choice to represent a complacent model, not a measurement of any real model.

# A scripted "model" proposes two versions of a function that must sum the EVEN numbers of a list.
CANDIDATES = [
    "def sum_even(xs):\n    return sum(xs)\n",                      # wrong: sums everything
    "def sum_even(xs):\n    return sum(x for x in xs if x % 2 == 0)\n",
]
TESTS = [([1, 2, 3, 4], 6), ([], 0), ([7, 9], 0)]


def self_critique(code):
    """Script representing a complacent model: it always finds its own work correct."""
    return "ok"


def external_test(code):
    environment = {}
    exec(code, environment)   # only because the code is ours; a model's code must be isolated (article 10)
    errors = [(x, expected, environment["sum_even"](x)) for x, expected in TESTS
              if environment["sum_even"](x) != expected]
    return errors


print("--- accept if the self-critique says ok")
for n, code in enumerate(CANDIDATES, start=1):
    if self_critique(code) == "ok":
        print(f"attempt {n}: accepted; errors in the tests = {len(external_test(code))}")
        break

print("--- accept if the external tests pass")
for n, code in enumerate(CANDIDATES, start=1):
    errors = external_test(code)
    if not errors:
        print(f"attempt {n}: accepted")
        break
    print(f"attempt {n}: rejected, {len(errors)} tests failed, the first {errors[0]}")

# Output:
# --- accept if the self-critique says ok
# attempt 1: accepted; errors in the tests = 2
# --- accept if the external tests pass
# attempt 1: rejected, 2 tests failed, the first ([1, 2, 3, 4], 6, 10)
# attempt 2: accepted

The first loop accepts the first version, which fails two tests out of three. The second rejects it, shows the failed case (the sum of all the numbers, 10, instead of 6) and accepts the second. The code has a note: exec runs the strings in the current process only because I wrote them; code produced by a model must be isolated, as the article on security says. The criterion for your agents: if an external check exists (tests, validation, another source), put it in the loop; if it does not exist, a second pass of the model on itself should not be considered a guarantee.

Long context and invented answers

Two limits that the series has met several times have an explanation in research. On context, Liu and colleagues (TACL, 2023) analyzed multi-document question answering and key-value retrieval: performance can degrade significantly when the position of the relevant information changes, with the best results at the beginning or the end of the context and worse results when the information is in the middle, even in models designed for long contexts. Direct consequence for an agent that accumulates history: it is not enough that the datum is in the window, where it sits matters (articles on memory and on costs).

On invented answers, Kalai and colleagues (2025) argue that models "hallucinate" because training and evaluation procedures reward guessing over acknowledging uncertainty, and that the remedy also goes through how scores are assigned to benchmarks. For whoever builds, the useful reading is another: if the evaluation rewards answering anyway, your system must have an explicit way out for "I don't know", like the report without a conclusion in the constructed example, and a check that respects it.

Work and adoption: two studies, two results

Many percentages circulate on work; two studies with a declared method show why they should be read with caution. Brynjolfsson, Li and Raymond (NBER, 2023) study the staggered introduction of a conversational assistant with data from 5,179 customer support agents: productivity, measured in issues resolved per hour, rises on average by 14%, by 34% for less experienced workers, with minimal impact on experts. The work of the METR group (2025) is a randomized trial on 16 experienced open-source developers and 246 tasks in projects they had known for years: the developers forecast a 24% reduction in time, and instead took 19% more time when AI tools were allowed.

They are not in conflict: they measure different jobs, tools, experience levels and periods. The point is that neither says what will happen in your department, and that the same developers' forecast (24% faster) had the wrong sign. Experts' forecasts also vary greatly: in the survey of 2,778 AI researchers (Grace and colleagues, 2024) the aggregate gives a 10% chance by 2027 and 50% by 2047 that unaided machines outperform humans in every possible task, 13 years earlier than the survey of the previous year, while for the full automation of all occupations the 50% arrives at 2116. Before trusting a forecast, ask four questions:

• Who measured, and what: an interview, sales data or a controlled trial have different weights.
• Which metric: exact match, time, the satisfaction of those who use the tool tell different things.
• Against what baseline: "faster" than whom, and on which tasks.
• Who gains if the forecast is believed: a vendor, a market analyst, a researcher with funding.

A figure that does not answer these questions is an opinion. In practice it means that decisions about your agent are made with a small trial of your own (one department, one month, a comparison with and without) and not with an industry average. The metrics introduced in the article on deployment serve this purpose too.

The European regulatory framework: what the text says

For those who operate in Europe, the reference is Regulation (EU) 2024/1689 on artificial intelligence (AI Act), published in the Official Journal on 12 July 2024. What follows is not legal advice.

The application calendar of Regulation (EU) 2024/1689 (AI Act) was amended by Regulation (EU) 2026/1744 (OJ L, 24.7.2026, in force from 27.7.2026). As of today: prohibitions and Article 4 from 2 February 2025; general-purpose AI models, governance and penalties from 2 August 2025; general application and Article 50 from 2 August 2026; Annex III high-risk from 2 December 2027; Annex I high-risk from 2 August 2028. Dates read on EUR-Lex on 2026-11-13; check the consolidated version before setting a deadline. Article 113 of the 2024 text said the regulation applies from 2 August 2026, with exceptions: Chapters I and II from 2 February 2025, some parts (including the obligations for general-purpose models and governance) from 2 August 2025, and Article 6(1) from 2 August 2027; the last two high-risk dates were moved as above.

For an agent, the classification and three articles matter. A system is high-risk under Article 6 if it is a safety component of a regulated product (Annex I) or falls within the uses of Annex III, except for the derogation in paragraph 3 for cases without significant risk: the use counts, not the fact that the system is "agentic". For high-risk systems Article 14 asks for effective human oversight, with measures commensurate with the risks, the level of autonomy and the context, and wants those who oversee to be able to remain aware of the tendency to rely automatically on the output. Article 50 requires providers of systems intended to interact directly with people to ensure they are informed that they are interacting with an AI system, unless this is obvious.

Translated into design choices: the boundary between proposing and acting (permissions, human confirmation after external content, caps) is the same structure the article on security builds for technical reasons, and the action log of that same article is what lets a person truly oversee. If your use falls under Annex III, ask for an opinion from someone qualified to give it; do not let a listing decide.

When not to use an agent: the summary of the series

After fourteen articles the most useful question remains the first: do you really need an agent? The table gathers the criteria that emerged, each with the article where it is demonstrated with a listing.

Each row is a condition, not an absolute recommendation: an agent passes if all of them hold for your task, and almost never for all tasks. The constructed case of article 13 shows how the rows become a graph, a cap and a verifier.

Frequently asked questions

What you can do now

You can compute how well a chain of steps holds and what a check with repetition gains, recognize when an "emergent capability" depends on the metric, and put an external signal in the loop instead of relying on the model's self-critique. You can read a forecast with four questions, knowing that two serious studies on work gave opposite results, and you know where the European regulation speaks of human oversight and transparency. Above all you can decide whether a task deserves an agent, and reread the series from the point you need: from the definition to the deployment.