AgentFlowAcademy

Training an LLM

An assistant like Gemini is built in stages. Pre-training on huge amounts of text makes a base model, fine-tuning teaches it to follow instructions, and human feedback shapes what a good answer looks like.

9 min Free

An LLM isn’t programmed. It’s trained: its billions of weights start random and are adjusted, step by step, until it predicts text well. That’s the same loop you saw in How a neural network learns, at a vastly larger scale.

Chat assistants are usually trained in three stages:

  • Pre-training. The model learns to predict hidden tokens in a very large, filtered text dataset. For text generators, that means predicting the next token. This is by far the most expensive stage: Google’s crash course says it can take months, with huge amounts of compute and electricity. The result is a base model (also called a foundation model).
  • Fine-tuning. Extra training on a much smaller set of examples for a specific job. Instruction tuning is fine-tuning on prompts paired with good responses, so the model learns to follow instructions.
  • Learning from human feedback. People compare answers, a reward model learns their preferences, and reinforcement learning pushes the LLM towards answers people prefer. This is called RLHF (reinforcement learning from human feedback).

Press Play to follow one model through all three. Then switch Try it to Same question: base model vs tuned model.

  1. Pre-train
  2. Fine-tune
  3. Human feedback
  4. Use
Web texthuge text datasetExample chatsprompt + ideal replyHuman raterspick the better oneTraining looploss + backpropReward modelpredicts preferenceThe LLMbillions of weightsYouafter training

Press Play to watch it run, or step through with ← →.

00/14

Training log

Nothing trained yet. Press Play.

What to notice

  1. Knowledge mostly comes from pre-training. Facts, grammar, languages and coding patterns are all picked up while predicting the next token. Later stages mostly change behaviour, not knowledge.
  2. A base model completes text; it doesn’t answer. Ask it a question and it may continue with another question. Fine-tuning teaches it what should follow a request.
  3. Fine-tuning is the same loop with different data. Same loss, same backpropagation. Only the examples change, and there are far fewer of them.
  4. RLHF learns from preferences, not correct answers. For “write a polite refund reply”, there’s no single right text. It’s easier for people to say “A is better than B”, and the reward model turns those choices into a score.
  5. Using a model doesn’t train it. When you chat with an LLM, its weights stay the same. It “remembers” earlier messages only because they’re sent again in the prompt. See Context windows.

What you can change as a builder

Prompt engineering

Change the instructions and examples you send. No training at all, and the model’s parameters don’t change. Always try this first.

Fine-tuning

Train a foundation model further on examples of your task. The model keeps the same number of parameters but behaves more like your examples.

Distillation

Use a big model to label data, then train a smaller, cheaper model on it. Faster predictions, usually slightly less accurate.

Check your understanding

1. Which stage gives an LLM most of its general knowledge?
2. In RLHF, what do the human raters actually do?
3. A user tells your support bot their name. Next week, in a new chat, the bot doesn't know it. Why?