Why One Neuron Isn't Enough: The XOR Problem and the Case for Hidden Layers

In the last post we taught a single neuron to find its own weights. It started from random numbers and, by following gradients downhill, ended up very close to the hidden teacher weights. That felt like real progress, so it is a good moment to ask an uncomfortable question: is one neuron enough?

It is not. In this post we will see exactly where a single neuron breaks, prove that the problem is not bad luck or bad training, and discover why the answer to it is the idea behind every neural network: layers.

What a single neuron really computes

Strip away the code and a neuron computes this:

z = w1*x1 + w2*x2 + ... + b
output = sigmoid(z)

Sigmoid returns more than 0.5 exactly when z is greater than 0, and less than 0.5 when z is negative. So if we turn the output into a yes/no decision, the neuron answers "yes" when w1*x1 + w2*x2 + b > 0 and "no" otherwise.

The boundary between yes and no is the set of points where w1*x1 + w2*x2 + b = 0. With two inputs, that is a straight line. With three inputs it is a flat plane, and with more inputs a flat "hyperplane". The neuron cuts the space of inputs into exactly two halves with one flat cut, and everything on one side gets "yes".

The weights decide the direction of that cut, and the bias slides it back and forth. Training moves the line around until it separates the examples as well as it can. Sigmoid does not change this: it only makes the answer smooth near the line instead of abrupt.

This is a powerful tool when the data can be split by a straight line. The trouble starts when it cannot.

A tiny test bench: logic gates

The cleanest way to see the limit is with the smallest possible data: two inputs that are each either off (0) or on (1), giving four combinations. We will teach the neuron three classic logic gates.

# inputs: every combination of two on/off signals
X = np.array([
    [0, 0],
    [0, 1],
    [1, 0],
    [1, 1],
], dtype=float)

and_targets = np.array([0, 0, 0, 1], dtype=float)   # 1 only if both are on
or_targets  = np.array([0, 1, 1, 1], dtype=float)   # 1 if at least one is on
xor_targets = np.array([0, 1, 1, 0], dtype=float)   # 1 if exactly one is on

• AND is 1 only when both inputs are on.
• OR is 1 when at least one input is on.
• XOR ("exclusive or") is 1 when exactly one input is on. If both are on, or both are off, the answer is 0.

x1	x2	AND	OR	XOR
0	0	0	0	0
0	1	0	1	1
1	0	0	1	1
1	1	1	1	0

We reuse the exact SingleNeuron class from the last post (sigmoid activation, squared-error loss, backpropagation), and train it on each gate for 10,000 epochs:

def train(targets, epochs=10000):
    neuron = SingleNeuron(n_inputs=2, learning_rate=0.5)   # the class from the last post
    for epoch in range(epochs):
        total_loss = 0.0
        for x_i, target_value in zip(X, targets):
            predicted_value = neuron.forward(x_i)
            total_loss += neuron.loss(predicted_value, target_value)
            neuron.backward_pass(x_i, predicted_value, target_value)
    return neuron, total_loss / len(X)

AND and OR: the neuron succeeds

Here is what the neuron learned for the first two gates:

AND  weights [7.285 7.285]  bias -11.014  predictions [0.0, 0.023, 0.023, 0.972]
OR   weights [7.944 7.944]  bias  -3.736  predictions [0.023, 0.985, 0.985, 1.0]

For AND and OR, the predictions are very close to the targets: the AND neuron outputs 0.972 for the (1, 1) case and under 0.03 for the rest, and the OR neuron outputs under 0.03 for (0, 0) and above 0.98 for the others.

The weights even make sense. For AND, both weights are about 7.3 and the bias is about -11. The weighted sum only gets above zero when both inputs are on (7.3 + 7.3 - 11 = 3.6), while one input alone gives 7.3 - 11 = -3.7, which is negative. For OR, the bias is smaller in size (-3.7), so a single active input (7.9 - 3.7 = 4.2) is enough to push the sum above zero.

In both cases a single straight line can put the "1" points on one side and the "0" points on the other.

XOR: the neuron fails

Now look at what happened for XOR:

XOR  weights [-0.129 -0.064] bias  0.064  predictions [0.516, 0.5, 0.484, 0.468]

XOR average loss
epoch     0 | 0.1424
epoch   100 | 0.1332
epoch  1000 | 0.1332
epoch  9999 | 0.1332

The predictions are all close to 0.5. The neuron is saying "I have no idea" for every input. The loss stops improving after about 100 epochs and stays flat at around 0.133, which is close to the 0.125 you would get by simply answering 0.5 to everything. Ten thousand epochs make no difference.

The picture shows why. The two points that should give 1 sit at opposite corners, and so do the two points that should give 0. Whatever line you draw, you will always put at least one point on the wrong side. Try it with a pencil: a diagonal line separates one corner but cuts off a point of the wrong kind with it, and a line that splits the grid down the middle puts one of each kind on both sides.

Data like this is called not linearly separable: no straight line (or flat plane) can separate the classes.

This is not a training problem

It is tempting to blame the learning rate, the random seed, or the number of epochs. Those are not the cause. We can prove that no choice of weights at all can solve XOR with one neuron.

For the neuron to be correct, the weighted sum z must be negative for the rows that should give 0, and positive for the rows that should give 1:

(0,0) -> 0 :  b < 0                 (z must be negative)
(0,1) -> 1 :  w2 + b > 0
(1,0) -> 1 :  w1 + b > 0
(1,1) -> 0 :  w1 + w2 + b < 0

Now add the middle two conditions together:

(w1 + b) + (w2 + b) > 0          adding the two "1" rows
w1 + w2 + 2b > 0
w1 + w2 + b > -b                  subtracting b from both sides
w1 + w2 + b > 0                   because b < 0, so -b is positive

but row four demands:  w1 + w2 + b < 0      contradiction

The four conditions cannot all be true at once. It does not matter which numbers we try, or how long we train: no valid weights exist. This is an important distinction in machine learning. Sometimes a model fails because we have not trained it well enough. Here the model is simply too limited in what it can express, and more training cannot fix that. In the older literature this was a famous result, and it contributed to a period when interest in neural networks cooled down.

The obvious fix that doesn't work: stack more neurons

If one neuron can only draw one line, why not add a second neuron after it, so one neuron feeds another? Surprisingly, if we do that without a non-linear activation function in between, nothing improves. Here is the algebra for two layers with no activation between them:

layer2( layer1(x) ) = W2 * (W1 * x + b1) + b2
                    = (W2 * W1) * x + (W2 * b1 + b2)
                    = W * x + b

The result, W * x + b, has exactly the same shape as one neuron. Two linear layers collapse into a single linear layer with merged weights. We can check it in NumPy:

# layer 1: 2 inputs -> 2 neurons       layer 2: 2 neurons -> 1 neuron
W1 = np.array([[1.0, 2.0], [3.0, -1.0]])
b1 = np.array([0.5, -1.0])
W2 = np.array([2.0, -1.0])
b2 = 0.3

x = np.array([1.0, 2.0])

# two layers, no activation function in between
two_layers = W2 @ (W1 @ x + b1) + b2

# one layer with merged weights
W = W2 @ W1            # [-1.  5.]
b = W2 @ b1 + b2       # 2.3
one_layer = W @ x + b

print(two_layers, one_layer)   # 11.3 11.3

(The @ operator is matrix multiplication, which we will use more and more.) Both ways give 11.3, because they are mathematically the same thing. Stacking linear layers a hundred deep would still give just one line.

The ingredient that makes layers worth it: non-linearity

This is the real reason for activation functions, which we met in the first post. When each layer applies a non-linear function like sigmoid or ReLU to its output, the layers can no longer be merged into one. Each layer bends the space a bit, and the bends add up.

Now a network can do something a single neuron cannot: the first layer can transform the inputs into a new representation, and the second layer can draw a line in that new space.

Solving XOR with a hidden layer

Look at the XOR table again. XOR means "OR, but not AND": it is on when at least one input is on, but not when both are. We already have neurons that compute OR and AND. So the plan is:

• Hidden neuron 1 computes something like OR.
• Hidden neuron 2 computes something like AND.
• The output neuron fires when neuron 1 is on and neuron 2 is off.

Layers between the inputs and the output are called hidden layers, because we do not directly see their values in the data or the answer. For now we will set the weights by hand, just as we did in the first post, so that we can see the idea clearly before we worry about training it:

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Hidden layer: 2 inputs -> 2 hidden neurons (each COLUMN is one neuron)
W1 = np.array([[20.0, 20.0],
               [20.0, 20.0]])
b1 = np.array([-10.0, -30.0])

# Output layer: 2 hidden neurons -> 1 output neuron
W2 = np.array([20.0, -20.0])
b2 = -10.0

hidden = sigmoid(X @ W1 + b1)          # shape (4, 2): one row per example
output = sigmoid(hidden @ W2 + b2)     # shape (4,)

print(hidden.round(3))
print(output.round(3))

How do these numbers work? The large weights (20) make sigmoid behave almost like an on/off switch:

• Hidden neuron 1 computes 20*x1 + 20*x2 - 10. If both inputs are off, that is -10 and the neuron outputs about 0. If one input is on, it is +10 and the output is about 1. So it behaves like OR.
• Hidden neuron 2 computes 20*x1 + 20*x2 - 30. One input on gives -10 (output about 0), and only both on gives +10 (output about 1). So it behaves like AND.
• The output neuron computes 20*h1 - 20*h2 - 10. It is positive only when h1 is 1 and h2 is 0, which is exactly "OR but not AND".

Running it prints (rounded):

hidden (rounded)      output
[0, 0] -> [0, 0]  ->  0.0
[0, 1] -> [1, 0]  ->  1.0
[1, 0] -> [1, 0]  ->  1.0
[1, 1] -> [1, 1]  ->  0.0

All four rows match XOR: 0, 1, 1, 0. A network that could not be built with one neuron now works with three.

Why it works: the hidden layer reshapes the space

Look at the middle column above. The original four points were arranged so that no line could separate them. After the hidden layer, they have new coordinates:

• (0, 0) became (0, 0)
• (0, 1) and (1, 0) both became (1, 0)
• (1, 1) became (1, 1)

The two points that should give 1 have been moved onto the same spot, and in this new space the points that should give 0 sit at (0, 0) and (1, 1). Now the classes are separable by a straight line: for example, a line that passes between (1, 0) and the other two. The output neuron is just a single neuron drawing that line, and it succeeds because the hidden layer rearranged the data to make that possible.

This is the main idea of deep learning in one sentence: the hidden layers learn a new way of describing the inputs that makes the final decision easy. Early layers build simple features, and later layers combine them into more complex ones. In an image network, early layers may respond to edges, middle layers to shapes, and later layers to whole objects. In our tiny network, the "features" are OR and AND.

How far can this go?

With more hidden neurons, a network can combine many lines to build up curved and complicated boundaries, carving the space into regions of any shape. There is a famous theoretical result, the universal approximation theorem, which says that a network with even one hidden layer and enough neurons can approximate a very wide class of functions to any accuracy. In practice, adding depth often makes learning much more efficient than making a single layer extremely wide, which is why modern networks use many layers.

The new problem: how do we train it?

We solved XOR, but notice what we did: we picked all those weights by hand, using insight about OR and AND. That works for a puzzle with four rows. It would be impossible for a problem with thousands of features and millions of weights.

We need the network to find those weights itself, as our single neuron did in the last post. But now there is a complication. In the single neuron, each weight touched the output directly, so its gradient was simple. In a two-layer network, the weights in the first layer affect the output only through the hidden neurons. To train them, the error signal has to travel backward through the second layer to reach the first.

The chain rule we used last time extends naturally to this situation: the gradient for a hidden weight is a longer chain of multiplied derivatives. That is why the algorithm is called backpropagation: the error is propagated back, layer by layer. In the last post we only saw it travel one step.

Key takeaways

1. A single neuron makes a decision by one flat cut through the input space: a line in two dimensions, a plane in three. It can only solve linearly separable problems.
2. AND and OR are linearly separable, and our neuron learned them easily. XOR is not, and the neuron got stuck at predicting about 0.5 for everything.
3. For XOR, the failure is not bad training. We showed algebraically that no weights can work.
4. Stacking layers with no activation function in between gains nothing, because linear layers collapse into one.
5. Non-linear activations between layers are what make depth useful. A hidden layer re-represents the data so that the output neuron's single line is enough.
6. Our hand-built network of three neurons solves XOR perfectly, but the weights were chosen by hand. Training them requires backpropagation through more than one layer.

What's next

In the next post we will build a small network layer class in NumPy, replace the slow loops with matrix operations so that all four examples are processed at once, and then derive backpropagation through two layers. We will start the XOR network from random weights and watch it find a solution on its own.