A Single Neuron From Scratch With NumPy: Weights, Bias, and Activation Functions

Every neural network, from a tiny classifier to a large language model, is built from the same small piece: the neuron. Before we stack layers, compute gradients, or train anything, we need to understand what one neuron actually does.

This is the first post in the series Neural Network From Scratch Using Numpy Only. We will not use PyTorch, TensorFlow, or any machine learning library. Just Python and NumPy, so that nothing is hidden behind a function call. In this post we build a single neuron, run two examples through it by hand, and look at three activation functions.

What is a neuron?

An artificial neuron is a small function that takes several numbers as input and produces one number as output. It does that in three steps:

1. Multiply each input by its own weight.
2. Add the results together, then add a bias.
3. Pass that total through an activation function.

As a formula, with inputs x, weights w, and bias b:

output = activation( x1*w1 + x2*w2 + x3*w3 + b )

That is the whole idea. Everything else in deep learning is about arranging many of these, and finding good values for the weights and biases.

The example: predicting a student's outcome

To keep the numbers meaningful, our inputs describe a student with three features:

• hour_study: hours spent studying
• party_hours: hours spent partying
• shoe_size: the student's shoe size

Shoe size is there on purpose. It is a feature that should have nothing to do with the outcome, and we will see how a neuron can learn to ignore it.

In NumPy, two students become a 2D array. Each row is one student and each column is one feature:

import numpy as np

# [hour_study, party_hours, shoe_size]
inputs = np.array([
    [10, 2, 9],
    [5, 0, 8],
])

The full code

Here is the complete program. We will go through it piece by piece below.

import numpy as np
from typing import Literal

# [hour_study, party_hours, shoe_size]
inputs = np.array([
    [10, 2, 9],
    [5, 0, 8],
])

ActivationType = Literal["step", "sigmoid", "relu"]


class SingleNeuron:
    def __init__(self, activation_type: ActivationType = "relu") -> None:
        self.weights = np.array([5, -1, 0], dtype=float)
        self.bias = 1.0
        self.activation_type = activation_type

    def forward(self, single_input: np.ndarray) -> float:
        forward_measure = np.dot(single_input, self.weights) + self.bias
        return self.activation_function(forward_measure)

    def step_function(self, forward_measure: float) -> int:
        return 1 if forward_measure >= 0 else 0

    def sigmoid_function(self, forward_measure: float) -> float:
        return 1 / (1 + np.exp(-forward_measure))

    def relu_function(self, forward_measure: float) -> float:
        return max(0.0, forward_measure)

    def activation_function(self, forward_measure: float) -> float:
        if self.activation_type == "step":
            return self.step_function(forward_measure)
        if self.activation_type == "sigmoid":
            return self.sigmoid_function(forward_measure)
        return self.relu_function(forward_measure)


if __name__ == "__main__":
    neuron = SingleNeuron(activation_type="relu")
    print(neuron.forward(inputs[0]))   # 49.0  (10*5 + 2*-1 + 9*0 + 1)
    print(neuron.forward(inputs[1]))   # 26.0  (5*5 + 0*-1 + 8*0 + 1)

Weights and bias: the neuron's parameters

The constructor sets up the two things a neuron owns:

self.weights = np.array([5, -1, 0], dtype=float)
self.bias = 1.0

The weights say how much each input matters, and in which direction:

• 5 for study hours: a large positive weight. More studying pushes the output up strongly.
• -1 for party hours: a small negative weight. More partying pushes the output down.
• 0 for shoe size: zero influence. Whatever the shoe size is, it contributes nothing.

The bias is a constant added no matter what the inputs are. It shifts the neuron's output up or down, and it lets the neuron produce a non-zero value even when every input is zero. Here it is 1.0.

The weights are written with dtype=float so NumPy stores them as floating point numbers. Later, when we train the network, weights will be adjusted in tiny decimal steps, so they need to be floats from the start.

For now these values are chosen by hand. Training, the process that finds good weights automatically, comes later in the series.

The forward pass

Sending an input through the neuron to get an output is called the forward pass. It lives in the forward method:

forward_measure = np.dot(single_input, self.weights) + self.bias
return self.activation_function(forward_measure)

np.dot computes the dot product: it multiplies the input and weight vectors element by element and sums the results. Then we add the bias. The variable is called forward_measure here. In many textbooks you will see it called the pre-activation or z.

Let's compute it by hand for the first student:

inputs[0] = [10, 2, 9]
weights   = [ 5, -1, 0]

10 * 5  =  50
 2 * -1 =  -2
 9 * 0  =   0
-----------------
weighted sum = 48
bias         = +1
-----------------
forward_measure = 49

And for the second student:

inputs[1] = [5, 0, 8]
weights   = [5, -1, 0]

5 * 5   =  25
0 * -1  =   0
8 * 0   =   0
-----------------
weighted sum = 25
bias         = +1
-----------------
forward_measure = 26

Notice how the zero weight did its job. Student 1 has shoe size 9 and student 2 has shoe size 8, but multiplying by 0 erased both. If you change the shoe size to 45, the result stays exactly the same.

Why we need an activation function

The weighted sum is a linear calculation. If we only ever did linear calculations, then stacking many neurons would still collapse into one big linear calculation, and the network could never learn curved or complex patterns.

The activation function adds a non-linear twist after the sum. It also controls what kind of output the neuron gives: a yes/no answer, a probability-like value, or an unbounded positive number. Our class supports three.

Step function

def step_function(self, forward_measure: float) -> int:
    return 1 if forward_measure >= 0 else 0

The oldest and simplest idea. The neuron either fires (1) or it doesn't (0), depending on whether the sum reaches zero. It gives a clean yes/no answer, but it has a serious drawback for learning: it has no smooth slope. A tiny change in the weights usually changes nothing, then suddenly flips the output. That makes it hard to improve weights gradually, which is why modern networks rarely use it.

Sigmoid function

def sigmoid_function(self, forward_measure: float) -> float:
    return 1 / (1 + np.exp(-forward_measure))

Sigmoid squashes any number into the range between 0 and 1. Large positive values approach 1, large negative values approach 0, and zero maps to exactly 0.5. Because the output is smooth, it can be read as a confidence or a probability, and small weight changes cause small output changes.

• sigmoid(0) = 0.5
• sigmoid(2) is about 0.88
• sigmoid(-2) is about 0.12
• sigmoid(49) is practically 1.0

One practical note: for a very large negative input, np.exp(-forward_measure) becomes huge and can overflow. Our small examples are safe, but production code uses a numerically stable version of sigmoid.

ReLU function

def relu_function(self, forward_measure: float) -> float:
    return max(0.0, forward_measure)

ReLU stands for Rectified Linear Unit. If the input is negative it returns 0, otherwise it returns the input unchanged. It is simple, cheap to compute, and the default choice in most modern hidden layers. This is why our neuron defaults to "relu".

That is also why our first result is 49.0 rather than something between 0 and 1: ReLU does not squash positive values, it passes them straight through.

Choosing the activation

The class stores the choice as a string, restricted by Literal["step", "sigmoid", "relu"]. That type hint tells your editor which values are valid, and catches typos before you run anything. The dispatcher then picks the right function:

def activation_function(self, forward_measure: float) -> float:
    if self.activation_type == "step":
        return self.step_function(forward_measure)
    if self.activation_type == "sigmoid":
        return self.sigmoid_function(forward_measure)
    return self.relu_function(forward_measure)

If the string is anything other than "step" or "sigmoid", it falls back to ReLU.

Running it

With the default ReLU neuron, the two students give:

neuron = SingleNeuron(activation_type="relu")
print(neuron.forward(inputs[0]))   # 49.0
print(neuron.forward(inputs[1]))   # 26.0

Both sums were positive, so ReLU simply passed them through. To see the activations behave differently, try a student who studies little and parties a lot:

neuron = SingleNeuron(activation_type="sigmoid")

# 1 hour of study, 8 party hours, shoe size 10
student = np.array([1, 8, 10])
# 1*5 + 8*-1 + 10*0 + 1 = -2
print(neuron.forward(student))   # about 0.119

neuron = SingleNeuron(activation_type="relu")
print(neuron.forward(student))   # 0.0

neuron = SingleNeuron(activation_type="step")
print(neuron.forward(student))   # 0

The sum is -2. ReLU clips it to 0, the step function says 0 (not fired), and sigmoid returns a value near 0.12, a soft "probably not". Same sum, three different personalities.

What this neuron cannot do yet

Our neuron works, but it is not intelligent. Nothing has been learned: a person typed in the weights 5, -1, and 0 based on a guess about how studying and partying should matter. A real network has to find those numbers itself from data.

It also handles one input at a time. The inputs array holds two students, but we call forward separately on each row. NumPy can process the whole batch at once with matrix multiplication, and we will take advantage of that soon.

Key takeaways

1. A neuron computes a weighted sum of its inputs, adds a bias, and applies an activation function.
2. Weights control how much each input matters and in which direction. A weight of zero means the input is ignored.
3. The bias shifts the result regardless of the inputs.
4. The activation function adds non-linearity and shapes the output: step gives 0 or 1, sigmoid gives a value between 0 and 1, ReLU gives 0 or the sum itself.
5. The forward pass is just np.dot(inputs, weights) + bias followed by the activation.

What's next

In the next post we will go from one neuron to a whole layer of neurons, replace the single dot product with a matrix multiplication, and process a batch of students in one call. After that, we will tackle the big question: how does a network find good weights on its own?