How a neural network learns
A neural network starts with random weights and gets better by making a guess, measuring how wrong it was, and nudging every weight a little. Train a real tiny network and watch its loss fall.
A neural network is a stack of simple calculations. Each neuron multiplies its inputs by weights, adds a bias, and passes the result through an activation function that squashes it into a useful range. Weights and biases together are the network’s parameters. Gemini has billions of them. The network below has nine.
Nobody sets those parameters by hand. They start as random numbers, and training improves them with a loop of four moves:
- Forward pass: run an example through the network and get a prediction.
- Loss: measure how wrong the prediction was, as one number.
- Backpropagation: work backwards to find each weight’s gradient, which says which way that weight pushed the loss and how hard.
- Update: move every weight a small step in the direction that lowers the loss. This is gradient descent.
Press Play to watch two training steps in slow motion, then the rest at speed. When it finishes, drag the learning rate to 0.1 or 8 and train again. The last run stays on the chart as a dashed line, so you can compare.
- Forward
- Loss
- Backprop
- Update
Evening 1: humidity 0.9, clouds 0.8
- 0.9 / 0.8rain50%
- 0.8 / 0.2dry51%
- 0.2 / 0.9dry50%
- 0.1 / 0.3dry51%
- 0.7 / 0.9rain50%
- 0.4 / 0.4dry51%
This tiny network guesses whether it will rain on a Chennai evening from two numbers: humidity and cloud cover. Its 6 weights and 3 biases start as random guesses. Press Play to train it.
Changing the learning rate restarts training and keeps the last run as a dashed line. The forward pass, log loss, backpropagation and updates are the real maths. The six evenings and starting weights are made up.
What to notice
- One step changes every weight a little. No single example teaches the network the rule. Hundreds of small nudges, each from one example, add up to a network that predicts rain only when humidity and clouds are high.
- Loss is the only thing training looks at. The network doesn’t “understand” rain. It finds whatever weights make the loss small on the examples it was shown. That’s why the quality of training data matters so much.
- The gradient gives direction, the learning rate gives step size. Too small (0.1) and the loss crawls down. Too big (8) and each step overshoots, so the network gets stuck. Picking a good learning rate is part of the job.
- Hidden layers combine the inputs. Each hidden neuron learns its own mix of humidity and clouds, and the output neuron combines those mixes. The activation function (here, sigmoid) is what lets the network learn curved, non-linear patterns. Without it, stacked layers would still only draw straight lines.
- An LLM learns the same way, at a vast scale. Replace “rain or not” with “which token comes next”, six evenings with trillions of tokens, and nine parameters with billions. The loop is still forward, loss, backprop, update. See Training an LLM.