45 - The perceptron and activation functions: what non-linearity is for

A nerve cell receives signals from many others, adds them up somehow and, if the sum exceeds a certain threshold, "fires" and sends a signal of its own. This very simplified picture inspired the psychologist Frank Rosenblatt, in 1958, to build the perceptron, the first artificial neuron that learned from data. An honest caveat is due: the artificial neuron is a metaphor inspired by biology, not a simulation of it; what we use today is a very simple mathematical function. But this function, repeated thousands of times and wired in layers, is the building block of every neural network.

In this lesson you build the neuron from scratch, watch it learn with a very simple rule, discover the limit that stopped research for years in 1969 (it cannot learn XOR) and grasp the most important lesson of the whole module: without a non-linear activation function, a stack of layers is equivalent to a single layer.

The artificial neuron

A neuron has three ingredients: the inputs x₁, ..., x_n; a weight w_j for each input, which says how much that input counts; and a bias b, a number that shifts the activation threshold (the same role as the intercept in regression). The neuron computes two things in sequence:

z = w₁x₁ + w₂x₂ + ... + w_n x_n + b = w · x + b          (weighted sum, also called pre-activation or logit)
a = f(z)                                                  (activation: the function f decides the output)

If you recognize the formula, that is normal: z is the same linear combination as in linear regression and logistic regression. The difference lies in the choice of f. With f(z) = z (no activation) the neuron is a linear regression. With f = sigmoid it is a logistic regression: a logistic regression is a neuron with a sigmoid activation, as anticipated in the previous lesson. With f = step you get the original perceptron:

f(z) = 1  if z ≥ 0,   0  otherwise

The perceptron learning rule

Rosenblatt proposed a simple update rule. For each example (x, y) with y ∈ {0, 1}: compute the prediction ŷ = f(w·x + b); if it is right, do nothing; if it is wrong, move the weight in the direction that brings the prediction closer to the true value:

w ← w + η · (y − ŷ) · x
b ← b + η · (y − ŷ)

η (eta) is the learning rate, a positive number that sets the size of the shift. Why does it work? If y = 1 and ŷ = 0 (the neuron said "no" but should have said "yes"), the update is w ← w + η·x. The new score on the same example becomes (w + ηx)·x + b + η = (w·x + b) + η(‖x‖² + 1), that is, it grows by a positive amount: we are pushing z toward the right side of the threshold. If y = 0 and ŷ = 1, the sign is opposite and z decreases. Examples that are already correct change nothing. This is a very simple special case of gradient descent.

There is a theoretical guarantee, the perceptron convergence theorem (Novikoff, 1962): if the data are linearly separable (there is a hyperplane that separates the two classes perfectly), the rule finds a separator in a finite number of updates. If they are not separable, it never converges. We check this with code.

Hand-written implementation: AND, OR and the XOR wall

Logic functions with two binary inputs are the ideal test bench: they are 4 points in a plane, and you can see at once whether a line exists that separates the 1s from the 0s.

import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)

def perceptron(X, y, eta=1.0, epochs=100):
    w, b, errors_per_epoch = np.zeros(2), 0.0, []
    for _ in range(epochs):
        errors = 0
        for xi, yi in zip(X, y):
            pred = 1 if w @ xi + b >= 0 else 0                 # step function
            if pred != yi:
                w = w + eta * (yi - pred) * xi                   # perceptron rule
                b = b + eta * (yi - pred)
                errors += 1
        errors_per_epoch.append(errors)
        if errors == 0:                                          # an epoch with no errors: convergence
            break
    return w, b, errors_per_epoch

for name, y in (("AND", [0, 0, 0, 1]), ("OR", [0, 1, 1, 1]), ("XOR", [0, 1, 1, 0])):
    w, b, errors = perceptron(X, np.array(y))
    print("%-4s  epochs used: %3d   errors per epoch (last 3): %s   w = %s  b = %.0f" % (name, len(errors), errors[-3:], w, b))

AND   epochs used:   6   errors per epoch (last 3): [2, 1, 0]   w = [2. 1.]  b = -3
OR    epochs used:   4   errors per epoch (last 3): [2, 1, 0]   w = [1. 1.]  b = -1
XOR   epochs used: 100   errors per epoch (last 3): [4, 4, 4]   w = [-1.  0.]  b = 0

What the code does. perceptron goes through the examples one at a time (an epoch is one pass over all of them), applies the step, and if the prediction is wrong it updates weights and bias with the rule. It stops at the first pass with no errors. For AND and OR it converges in a few epochs: the four points can be separated with a line. For XOR (output 1 only when the two inputs differ) the perceptron uses all 100 epochs it is granted and keeps getting it wrong: no line separates {(0,1), (1,0)} from {(0,0), (1,1)}, because the two diagonals of the square cross. In 1969 Marvin Minsky and Seymour Papert rigorously proved this limitation of single-layer perceptrons, and for many years enthusiasm for neural networks cooled.

The way out: more neurons, in two layers

XOR is solved with two layers. The idea: instead of a single line you need two, whose results are combined. We build the solution by hand, choosing the weights by reasoning, and then check it:

The construction has three neurons, and the order matters because the third reads the outputs of the first two. Neuron h₁ computes OR(x₁, x₂): it fires if the sum of the inputs is at least 0.5, that is step(x₁ + x₂ − 0.5). Neuron h₂ computes AND(x₁, x₂): it fires if the sum is at least 1.5, that is step(x₁ + x₂ − 1.5). The output says "OR but not AND", that is step(h₁ − h₂ − 0.5), which is exactly XOR.

step = lambda z: (z >= 0).astype(int)

h1 = step(X @ np.array([1, 1]) - 0.5)              # OR
h2 = step(X @ np.array([1, 1]) - 1.5)              # AND
output = step(h1 - h2 - 0.5)                       # OR and not AND

print("x1 x2 | h1(OR) h2(AND) | XOR output")
for xi, a, c, o in zip(X.astype(int), h1, h2, output):
    print(" %d  %d |   %d      %d     |    %d" % (xi[0], xi[1], a, c, o))

x1 x2 | h1(OR) h2(AND) | XOR output
 0  0 |   0      0     |    0
 0  1 |   1      0     |    1
 1  0 |   1      0     |    1
 1  1 |   1      1     |    0

What the code does. X @ np.array([1, 1]) computes x₁ + x₂ for each row (matrix-vector product); subtracting a constant is the bias. The hidden layer (h₁, h₂) remaps the four points into a new space where they are separable: the points (0,1) and (1,0) both become (1,0), while (0,0) becomes (0,0) and (1,1) becomes (1,1). In this new space a single line is enough. This is the fundamental idea of a deep network: the intermediate layers learn a representation of the data in which the problem becomes simple.

Non-linearity is not a detail: two linear layers are one layer

In the XOR solution every neuron uses the step, which is not linear. What would happen if the layers used f(z) = z? Stack two linear layers, with weight matrices W₁ (3×2) and W₂ (1×3): the output is W₂(W₁x) = (W₂W₁)x. The product W₂W₁ is a single (1×2) matrix: the whole network is equivalent to a single linear layer, however many layers you stack. The associativity of matrix multiplication cancels all the depth. Numerical check:

rng = np.random.default_rng(0)
W1 = rng.normal(size=(3, 2))                       # first layer: 2 inputs -> 3 neurons
W2 = rng.normal(size=(1, 3))                       # second layer: 3 -> 1
x = rng.normal(size=2)

a_two_layers = W2 @ (W1 @ x)
a_one_layer = (W2 @ W1) @ x
print("W2 (W1 x) == (W2 W1) x ?", np.allclose(a_two_layers, a_one_layer))
print("shape of W2 @ W1:", (W2 @ W1).shape, " -> a single 1x2 layer")

W2 (W1 x) == (W2 W1) x ? True
shape of W2 @ W1: (1, 2)  -> a single 1x2 layer

The code shows that the network with two linear layers and the single layer with matrix W2 @ W1 produce the same result. This is why non-linear activation functions are needed between the layers: they break the composition property and let the network represent functions that no linear model can imitate.

The activation functions

The original perceptron uses the step, but there is a fatal problem for modern learning: its derivative is zero everywhere (except at z = 0, where it does not exist), so gradient descent has no signal to follow. Smooth or nearly smooth functions are used instead:

• Sigmoid σ(z) = 1/(1 + e^(−z)): output in (0, 1). Derivative σ(1 − σ) with a maximum of 0.25. It saturates: for large |z| the derivative is almost zero.
• Hyperbolic tangent tanh(z): output in (−1, 1), centered on zero (convenient for the following layers). Derivative 1 − tanh²(z), with a maximum of 1 at z = 0. It saturates like the sigmoid.
• ReLU (Rectified Linear Unit) f(z) = max(0, z): zero for negative z, identity for positive z. Derivative 0 or 1. It does not saturate in the positive region and is cheap to compute: it is the default choice for hidden layers. Drawback: if a neuron always receives negative z, its derivative is always 0 and it stops learning ("dying ReLU"). The Leaky ReLU variant uses a small slope (for example 0.01) for negative z, so as not to vanish completely.

The three entries are read together by looking at the derivative, because the derivative decides whether the error signal crosses the layer. Sigmoid and tanh have a derivative that vanishes on both sides: the deeper the network, the more times the gradient passes through a flat zone. ReLU has a derivative of exactly 1 for positive z, so it does not attenuate the signal, but it zeroes it for negative z: the cost is the risk of neurons that never recover.

None of the three is "the best" in absolute terms; ReLU is the starting point because its flaws are easier to diagnose. The book by Goodfellow, Bengio and Courville describes ReLU as a piecewise linear function with two linear pieces, close to linear models and therefore easy to optimize (chapter 6, in the sources).

Let us measure the saturation.

sigmoid = lambda z: 1 / (1 + np.exp(-z))

print("   z   sigmoid deriv.   tanh deriv.   ReLU deriv.")
for z in (0.0, 2.0, 5.0, -5.0):
    print("%4.1f   %14.5f   %11.5f   %11.1f" % (z, sigmoid(z) * (1 - sigmoid(z)), 1 - np.tanh(z) ** 2, float(z > 0)))
print("\nproduct of 10 sigmoid derivatives in the best case (0.25 each): %.2e" % (0.25 ** 10))

   z   sigmoid deriv.   tanh deriv.   ReLU deriv.
 0.0          0.25000       1.00000           0.0
 2.0          0.10499       0.07065           1.0
 5.0          0.00665       0.00018           1.0
-5.0          0.00665       0.00018           0.0

product of 10 sigmoid derivatives in the best case (0.25 each): 9.54e-07

What the code does. It computes the three derivatives at four points. The sigmoid never exceeds 0.25, and at z = ±5 it equals 0.0066; tanh at z = ±5 equals 0.00018. ReLU equals 1 for positive z and 0 for negative z. The last line is the key: if a signal (in the direction of the gradient computation) has to cross 10 sigmoid layers, it is multiplied by 10 derivatives, each at most 0.25: in the best case 0.25¹⁰ ≈ 9.5·10⁻⁷. The gradient reaching the first layers is almost zero and those layers stop learning. This is the vanishing gradient problem, one of the reasons ReLU changed the practice of deep networks.

Which activation where

• Hidden layers: ReLU as a starting choice; Leaky ReLU or variants if you notice "dead" neurons; tanh in some contexts (recurrent networks).
• Output for binary classification: sigmoid (a probability).
• Output for multiclass classification: softmax (K probabilities that sum to 1).
• Output for regression: no activation (linear), because the value to predict has no bounds. If it is always positive you can use a ReLU or a softplus.

The rule behind the list is a single one: the activation of the output layer depends on what you want to predict (an unbounded number, a probability, a distribution over K classes), while that of the hidden layers depends on how the gradient behaves. This is why the two choices are made separately. The output is not chosen to optimize better: it is chosen to respect the meaning of the predicted value.

XOR learned from data, with a network written in NumPy

XOR can also be learned from data with gradient descent, without choosing the weights by hand. Here we write the 2 → h → 1 network directly in NumPy, with an activation between the layers and a binary cross-entropy loss, and we train it from 20 different random initializations to see how reliable it is. Writing the gradients by hand is instructive, because they are the ones that the backpropagation lesson derives in full; right after that, the same test is repeated with PyTorch, which computes them on its own.

sigmoid = lambda z: 1 / (1 + np.exp(-z))
yv = np.array([[0.0], [1.0], [1.0], [0.0]])
activations = {"Tanh": (np.tanh, lambda z: 1 - np.tanh(z) ** 2),
               "ReLU": (lambda z: np.maximum(0, z), lambda z: (z > 0).astype(float))}

def trial(h, name, seed, epochs=2000, lr=0.5):
    f, df = activations[name]
    rng = np.random.default_rng(seed)                                 # reproducible random initialization
    W1, b1 = rng.normal(size=(2, h)), np.zeros(h)
    W2, b2 = rng.normal(size=(h, 1)), np.zeros(1)
    for _ in range(epochs):                                           # gradient descent on the whole set (4 examples)
        z1 = X @ W1 + b1
        a1 = f(z1)
        z2 = a1 @ W2 + b2                                             # logit: the sigmoid is inside the loss
        dz2 = (sigmoid(z2) - yv) / len(X)                             # output minus target
        dz1 = (dz2 @ W2.T) * df(z1)                                   # chain rule through the hidden layer
        W2, b2 = W2 - lr * a1.T @ dz2, b2 - lr * dz2.sum(axis=0)
        W1, b1 = W1 - lr * X.T @ dz1, b1 - lr * dz1.sum(axis=0)
    z2 = f(X @ W1 + b1) @ W2 + b2
    loss = (np.logaddexp(0, z2) - yv * z2).mean()                     # stable binary cross-entropy
    return loss, bool(((z2 > 0).astype(float) == yv).all())

print("hidden neurons  activation   successes out of 20 initializations   median loss")
for h in (2, 4):
    for name in ("Tanh", "ReLU"):
        outcomes = [trial(h, name, seed) for seed in range(20)]
        print("%14d  %-12s %20d / 20   %22.4f" % (h, name, sum(c for _, c in outcomes), np.median([p for p, _ in outcomes])))

hidden neurons  activation   successes out of 20 initializations   median loss
             2  Tanh                            9 / 20                   0.3480
             2  ReLU                            2 / 20                   0.4775
             4  Tanh                           19 / 20                   0.0023
             4  ReLU                           13 / 20                   0.0019

The code has three parts. The forward pass computes z1, the activation a1 and the logit z2; the network returns the logit, the untransformed score, and not a probability, because the sigmoid is applied inside the loss (np.logaddexp(0, z2) - y*z2 is the binary cross-entropy written in a numerically stable form). If you applied a second sigmoid to the output and then used this loss, you would apply it twice. The backward pass starts from dz2 = output - target, the same quantity as in the section below, and multiplies it by the weights and by the derivative of the activation to reach the hidden layer: this is the chain rule. The update is the gradient descent of lesson 41.

The result is an honest message, and it holds for this initialization (standard normal weights, zero biases, 2000 steps with learning rate 0.5; measured with Python 3.14.4 and NumPy 2.3.5, two identical runs): with 2 hidden neurons the network learns XOR in 9 initializations out of 20 with tanh and in 2 out of 20 with ReLU; with 4 hidden neurons the numbers rise to 19 out of 20 (tanh) and 13 out of 20 (ReLU). The median loss with 2 neurons (0.35 and 0.48) is that of a network that in many trials stayed stuck, while with 4 neurons it is on the order of 0.002.

With few neurons, some initializations stay stuck in wrong solutions; more neurons offer more paths. With a different initialization the counts would change: that is exactly the point. Neural networks do not guarantee convergence, and the result depends on the initialization, the architecture and the learning rate. One limit of the experiment: the four XOR points alone do not allow any talk of generalization, and "success" only means that all four points are classified correctly.

The same network in PyTorch

Now the same test with the library: torch.nn.Sequential builds the 2 → h → 1 network, and backward() computes the gradients that you wrote by hand above. The listing was run with PyTorch 2.12.0 on CPU (CUDA_VISIBLE_DEVICES="", 2 threads, no GPU), twice, with identical output; the 160 trainings take from a few minutes to about ten, depending on the load of the machine. It uses the matrix X of the four points defined earlier.

import torch

Xt = torch.tensor(X, dtype=torch.float32)
yt = torch.tensor([[0.0], [1.0], [1.0], [0.0]])

def trial(h, activation, seed, epochs=2000, lr=0.5):
    torch.manual_seed(seed)                                                  # reproducible random initialization
    net = torch.nn.Sequential(torch.nn.Linear(2, h), activation(), torch.nn.Linear(h, 1))
    opt = torch.optim.SGD(net.parameters(), lr=lr)
    for _ in range(epochs):
        opt.zero_grad()
        loss = torch.nn.functional.binary_cross_entropy_with_logits(net(Xt), yt)   # the sigmoid is inside the loss
        loss.backward()
        opt.step()
    correct = bool(((net(Xt) > 0).float() == yt).all())
    return loss.item(), correct

print("hidden neurons  activation   successes out of 20 initializations   median loss")
for h in (2, 4):
    for activation in (torch.nn.Tanh, torch.nn.ReLU):
        outcomes = [trial(h, activation, seed) for seed in range(20)]
        print("%14d  %-12s %20d / 20   %22.4f" % (h, activation.__name__, sum(c for _, c in outcomes), np.median([p for p, _ in outcomes])))

hidden neurons  activation   successes out of 20 initializations   median loss
             2  Tanh                           11 / 20                   0.0041
             2  ReLU                            4 / 20                   0.6931
             4  Tanh                           20 / 20                   0.0024
             4  ReLU                           15 / 20                   0.0017

What the code does. nn.Sequential stacks a linear layer Linear(2, h) (the weight matrix and the bias), an activation and a second linear layer Linear(h, 1). The loss binary_cross_entropy_with_logits applies the sigmoid internally: this is why the network returns the logit and not a probability, and if you added a sigmoid to the output yourself you would apply it twice. backward() and step() are the ones from the gradient descent lesson. Compared with the NumPy version the initialization changes (Linear draws the weights from a uniform distribution of amplitude 1/√(inputs), not from a standard normal), and for this reason the counts do not coincide; the conclusion does. With 2 hidden neurons the network learns XOR in 11 initializations out of 20 with tanh and in 4 out of 20 with ReLU; with 4 neurons they rise to 20 out of 20 (tanh) and 15 out of 20 (ReLU).

The median loss 0.6931 of the ReLU case with 2 neurons is ln 2, the value of a network that predicts 0.5 for all four points: in at least half of the trials (it is a median) the network has learned nothing. Each of the two versions therefore has its role: the NumPy one shows the gradients, the PyTorch one shows how the library writes them; the counts of one are not to be read like those of the other.

The same object: linear regression, logistic regression and the perceptron

We have reached the point where it is worth stopping and looking at everything you have studied since the beginning of the module as a single thing. You have met three models with different names: linear regression, logistic regression and the perceptron. They look like distinct algorithms, with different cost functions and different update rules. In reality they are the same object, the neuron, in three configurations. The neuron always computes the score z = w·x + b. The three configurations differ only in two choices: the activation function f that turns z into the output a, and the cost function that measures how far a is from the target y.

There is, however, a surprising fact, which you will already have noticed while deriving the gradients in the previous lessons. For linear regression with squared cost, the gradient with respect to the weights for a single example is (a − y)·x. For logistic regression with cross-entropy, deriving the composition of sigmoid and logarithm, you had found (a − y)·x again. And Rosenblatt's rule, w ← w + η(y − ŷ)x, is the same thing written with the sign reversed: w ← w − η(ŷ − y)x.

Three models, a single update rule: weight ← weight − step × (output − target) × input. The logic is the same in all cases: the error (output minus target) is multiplied by the input that contributed to the error, and the weight moves accordingly. A large input contributes more, so its weight is corrected more; a zero input does not contribute, and its weight does not move.

Let us write it as a single program. The function below trains a neuron with stochastic gradient descent (one example at a time) and receives the activation function as an argument. For the three models only that argument and the learning rate change.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

Xc, yc = make_classification(n_samples=600, n_features=2, n_redundant=0, n_informative=2,
                             n_clusters_per_class=1, class_sep=1.2, flip_y=0.02, random_state=3)
Xtr, Xte, ytr, yte = train_test_split(Xc, yc, test_size=0.3, stratify=yc, random_state=0)
scaler = StandardScaler().fit(Xtr)                                     # parameters from the training set only
Xtr, Xte = scaler.transform(Xtr), scaler.transform(Xte)

identity = lambda z: z
sigmoid = lambda z: 1 / (1 + np.exp(-np.clip(z, -500, 500)))
step = lambda z: float(z >= 0)

def sgd_neuron(X, y, activation, eta, epochs, seed=0):
    rng = np.random.default_rng(seed)
    w, b = np.zeros(X.shape[1]), 0.0
    for _ in range(epochs):
        for i in rng.permutation(len(y)):                                # one example at a time, in random order
            a = activation(X[i] @ w + b)                                 # neuron output
            error = a - y[i]                                             # output minus target: identical in all three cases
            w = w - eta * error * X[i]
            b = b - eta * error
    return w, b

print("neuron        learning rate   learned weights           test accuracy")
for name, act, eta in (("linear", identity, 0.01), ("logistic", sigmoid, 0.1), ("perceptron", step, 0.1)):
    w, b = sgd_neuron(Xtr, ytr, act, eta, 30)
    pred = np.array([act(v) for v in Xte @ w + b]) >= 0.5
    print("%-12s  %13.2f   %-24s  %.3f" % (name, eta, np.round(np.r_[w, b], 3), (pred == yte).mean()))

neuron        learning rate   learned weights           test accuracy
linear                 0.01   [-0.438  0.03   0.498]    0.928
logistic               0.10   [-6.388 -0.138  1.583]    0.944
perceptron             0.10   [-0.452 -0.096  0.   ]    0.911

What the code does. The function sgd_neuron is the core: for each example it computes the output a by applying the activation to the score, computes error = a - y[i] and shifts weights and bias by eta * error * x. There is no "if" on the type of model: the difference lies entirely in the function passed as activation. The three neurons learn different but similar decision boundaries, all lines in the plane, and reach comparable accuracies on the test set: 0.928 for the linear neuron used as a classifier (threshold at 0.5 on the output), 0.944 for the logistic one and 0.911 for the perceptron.

The logistic neuron is the best of the three, which is consistent with the theory: it has a cost function designed for classification, and its weights are much larger (about −6.4 against −0.4) because the sigmoid needs large scores to push probabilities toward 0 and 1. The linear neuron, designed to predict numbers, can be used as a classifier but optimizes something slightly different from what you care about.

Why is this unified reading so important? For three practical reasons. First: it tells you what to choose. The output activation and the cost function are not two independent choices: they must be paired (identity with squared error, sigmoid with binary cross-entropy, softmax with categorical cross-entropy) because only with the right pairing does the gradient with respect to the score take the clean "output minus target" form you have seen here. Second: it tells you how to build a network.

A neural network is a set of these neurons arranged in layers, and the last layer is exactly one of the three objects in the table, chosen according to the problem: regression, binary or multiclass classification. Third: it tells you what awaits you in the next lessons. When you derive backpropagation, the formula that propagates the error backward starts from (output − target), the same quantity, and multiplies it layer after layer by the derivatives of the activations and by the weights.

Frequently asked questions, and the answers that avoid the most common misunderstandings

Does the artificial neuron work like a biological neuron? Only in a very vague way. A biological neuron is a cell with temporal dynamics, many chemical substances and complex connections. The artificial neuron is a function: a weighted sum followed by a non-linearity. The biological inspiration is historical and useful as a mental picture; it should not be taken as an explanation of the brain and should not guide the design choices of a network.

Why do I need the bias, if I already have the weights? Without a bias the score is w·x, and the decision boundary w·x = 0 always passes through the origin. If the data require a boundary that does not pass through the origin, the model has no way of representing it. The bias is the only parameter that is not multiplied by an input: it shifts the activation threshold without rotating the boundary. It is the same role as the intercept in linear regression.

If XOR is solved with two layers, are two layers enough for everything? No, and the precise answer is more interesting: a single hidden layer with enough neurons can approximate any continuous function (the universal approximation theorem, which you will see in the next lesson), but "enough" can mean an enormous number, and the theorem does not say how to find the weights nor does it guarantee that gradient descent finds them. In practice deeper networks with fewer neurons per layer are often more efficient to train and better at generalizing.

ReLU is a piecewise linear function: why does it work as a non-linearity? Because "piecewise linear" does not mean "linear". ReLU max(0, z) has a corner at z = 0, and a composition of piecewise linear functions with different corners can draw arbitrarily complex curves by approximating them with many segments. If there were no corner (that is, if you used f(z) = z) you would return to the collapse into a single linear layer that you verified earlier.

If the perceptron converges on AND and OR, why not always use the perceptron? Because it converges only if the data are linearly separable, and in reality they almost never are: there is noise, labeling errors, overlap between classes. On non-separable data the perceptron keeps oscillating between solutions without ever settling. Logistic regression, with a smooth cost function, finds instead the best possible boundary even when the classes overlap.

Is a network with more neurons always better? No. More neurons mean more capacity and therefore more risk of overfitting, more training time and more memory. The XOR experiment shows the other side: with 2 hidden neurons training succeeds only in some of the initializations, with 4 almost always. The right capacity is a compromise, chosen on validation.

Real-world traps

• Forgetting the activation between layers: the network becomes linear, and you do not notice until you look at its capacity. Fifteen linear layers are worth one.
• Sigmoid or tanh in deep hidden layers: vanishing gradient, training that stalls or is very slow.
• Dead ReLU: a learning rate that is too high or an unlucky initialization push many neurons into the negative zone, from which they do not return. Symptom: a large fraction of activations always at zero.
• Applying sigmoid/softmax twice: if the loss already includes the transformation (as in our logaddexp), the network must return the logits, not the probabilities.
• Initializing all weights to zero: the neurons of a layer become identical and receive the same gradients forever (symmetry). Initialize at random, with an appropriate scale.
• Expecting guaranteed convergence: as seen, it depends on the seed. An experiment repeated with several seeds is more honest than a single lucky run.
• Shape mismatch: X @ W1 with X of shape (4, 2) and W1 of shape (2, h) gives (4, h); passing a vector (2,) instead of a matrix (n, 2) works in NumPy but changes the meaning of the dimensions.

The seven traps have a common root: a network that is wrong in these ways keeps producing numbers, and makes no obvious errors. The network without activations and the one with all weights at zero train without complaining; the vanishing gradient shows up as a loss that does not go down. This is why the method of the XOR experiment applies: several trials with different seeds, an explicit check on weights and activations, and distrust of a single successful run.

What you can do now

You can build a neuron from scratch, train it with Rosenblatt's rule and explain why it cannot learn XOR. You can check numerically that two linear layers collapse into one, and choose the activation of the hidden layers and that of the output. You can also recognize linear regression, logistic regression and the perceptron as configurations of the same neuron with the same update rule. You have seen the previous lesson, overfitting and regularization; in the next one, layered networks, you stack many neurons and follow the shapes of the tensors in a complete forward pass.