KNOWLEDGE · LANGUAGE MODELS
Does AI learn while it answers? Pre-training, post-training, inference
One question tells the three apart: do the weights change? Pre-training and post-training are learning that changes the weights; inference leaves the weights as they are and only makes an answer from the text it is given (the context). What makes inference look like learning (pseudo-learning) happens only inside the context, and it is gone when the conversation ends.
TRY IT
Follow the smallest GPT
We trained a GPT with 27,648 weights ourselves. Pick a stage and press “One step” or “Play” to follow which layer the signal passes through. The eight characters on each layer are a fingerprint of that layer’s weights. If even one number changes, the fingerprint changes.
THE SAME QUESTION C, ONLY THE CONTEXT DIFFERS
All five rows use the same weights (same fingerprints). Change the example and the answer follows; with no example the model guesses evenly. It looks as if it learned from one look, but only the context changed. Press a row to walk through that context.
MODEL
- Outputnext-token probabilities4aec68f3
- Block 2 MLPper-token change68e9e8f3
- Block 2 attentionreads the context34f3909f
- Block 1 MLPper-token changed6773a34
- Block 1 attentionreads the contextbe238341
- Embeddingtoken → 32 numbers2c5adb54
Context (one text from the first batch, 32 tokens)
The first step of pre-training. The weights are still random. We feed in one of the 64 texts in the first batch. At every position the model must guess the very next token. Press “One step” and the signal climbs one layer at a time, starting from the embedding.
The 32 cells are the values that passed through the layer (the last token’s 32 numbers); the output’s 18 cells are the probabilities of the 18 tokens. positive negative, darker is larger. The yellow bar is the size of the gradient the layer received in the backward pass; the green bars under the context are the probability of guessing that token.
About this small GPT
- Structure. From the bottom up: token and position embeddings, two blocks (attention, then an MLP, in each), and the output layer. GPT is a language model built by stacking Transformer decoders this way. It has 27,648 weights, 18 tokens (
|start of a conversation,?question mark, keys A–H, values I–P), and a context of up to 32 tokens. - Why two blocks. The induction head, which sees [A][B] … [A] in the context and continues with [B], is a pair of attention heads in different layers, so it does not form in a one-layer model. When we trained both ourselves, one block got the value of a key called again right 28–31% of the time and two blocks 98–99%.
- What it was trained on. Pre-training ran 8,000 steps on text that lists key–value pairs and then calls the keys again (
|AKCNFJthenC N A K …), and on text that repeats the same letters. Post-training taught it to answer the example-free question| ? xwith a fixed mapping f(x). SFT got all eight questions right in 21 steps, reinforcement learning in 32. - Block 2’s lines. The last C in
|AKCNFJCgives 0.016 to the position of the answer N and 0.10–0.17 to the other positions. It barely looks at the answer’s position, yet it picks N. Unlike a textbook induction head, it may gather the positions that are not the answer and rule them out hypothesis. Still, switching off block 2’s attention drops accuracy on keys called again from 98.7% to 28.5%. Turn on “Switch off block 2 attention” in the comparison table above to see it. - The values. The pre-training stage starts from random weights; the post-training and inference stages start from the model that finished pre-training. What the training stages show (the forward pass of the random weights, probabilities, loss, gradients, fingerprints, drawn answers, curves) comes from re-running the first step and the record of the real training runs (8,000 pre-training steps, 300 RL steps) in Python. For the trained models, the next token and the attention are computed by this page with the same weights.
Sources: Induction heads: Olsson et al. 2022 · One-layer models: Elhage et al. 2021 · GPT’s structure: Radford et al. 2018 §3.1 · Small GPT: trained ourselves (Python, torch CPU)
01 PRE-TRAINING
Guessing the next token makes the weights in the first place
Show the model text and have it guess the next token at every position. A token is the unit the model handles text in (a word or a piece of a word). The loss is a number that is larger the lower the probability of the next token that actually came (−log p). Backpropagation finds a gradient for each weight, the weights are corrected a little, and this is repeated over an enormous amount of text. The weights are the numbers inside the model; everything the model knows and knows how to do is in them.
The first GPT paper split training into two stages: unsupervised pre-training, which learns a language model from a large body of text, then supervised fine-tuning, which fits the weights to a task’s right answers. The “pre” means the first of these two stages.
A model made this way continues text plausibly, but it has not been trained to answer the way people want. The skills it will use at inference also arise in this stage. The GPT-3 paper explains that the model picks up a broad set of skills and pattern recognition during training, and at inference uses them to adapt quickly to a task. The GPT-3 in that evaluation was only pre-trained, with no fine-tuning.
Sources: Radford et al. 2018 §3.1·§3.2 · Ouyang et al. 2022 abstract · Brown et al. 2020 §1, Figure 1.1, §2
02 POST-TRAINING
Training after pre-training
“Post” names a time, after pre-training; “training” means the weights change. It is not training done after release. Release means fixing the trained weights and putting the model out for people to use.
| Stage | What it fits | Loss |
|---|---|---|
| Pre-training | The next token in the text | −log p(next token) |
| SFT | Example good answers | −log p(example’s tokens) |
| Reinforcement learning | Reward (there is no right answer) | −(r − r) × log p(its own tokens) |
SFT (supervised fine-tuning) has the model copy example good answers. Reinforcement learning (RL) scores the answers the model writes itself and gets a reward r. The reward cannot be differentiated, so the loss is made as −(r − r) × log p. r is the average reward of the answers drawn for the same problem. The tokens of answers that did better than average become more likely, and those of answers that did worse become less likely. The reward can be the preference people chose (RLHF), or a right answer and format that rules can check.
DeepSeek-R1-Zero applied reinforcement learning straight to the only pre-trained DeepSeek-V3-Base, with no SFT. The longer it trained, the longer its answers became on average, and the paper wrote that the model learned by itself to solve problems by thinking longer. The machinery is the same as pre-training: loss → backpropagation → weights. What differs is what the loss is made of.
When it runs long on a narrow task
Other skills can shrink. InstructGPT called the drop in performance on public NLP tests after RLHF an “alignment tax”, and reduced the drop by mixing in updates that raise the log likelihood of the pre-training distribution (PPO-ptx). In the small GPT we trained too, teaching one mapping for 300 steps cut in-context learning from 99% to 73–75%.
Sources: Williams 1992 (policy gradient) · Shao et al. 2024 (GRPO) · Ouyang et al. 2022 §1, §3.5, §4.2 · DeepSeek-R1 paper §1, §1.1, §2.2.1, §2.2.2, §2.2.4, Figure 3 · Small GPT: trained ourselves (“About this small GPT” in Try it)
03 INFERENCE (PSEUDO-LEARNING)
The weights stay; it happens only in the context
At release the weights are fixed. Inference means putting a context into that model and drawing the next token one at a time to continue the answer. The context is all the text that went into the model in this conversation. The weights are only read.
Three things that look like learning
- It gets better after one look. Within one text, the model on this page guesses the value of a pair it sees for the first time only 13.9% of the time, but when that key is called again it gets the value right 98.7% of the time. The weights stay the same, yet it does better after seeing the pair once. How much lower the loss of later tokens in the context is than that of earlier ones is also a way to measure in-context learning.
- It changes to match what it is shown. The same question C is answered N (1.000) when the example in the context is C→N, L (0.951) when it is C→L, and P (0.999) when it is C→P. Asked in a new conversation with no example, it guesses evenly (P 0.119). In “Try it”, stage 03 Inference shows the five contexts side by side. The GPT-3 paper called this following of an unseen task from a few examples in-context learning, and wrote that no gradient updates and no fine-tuning take place.
- It reconsiders while answering. A reasoning model goes back over its earlier working and corrects it while writing an answer. The “aha moment” in which DeepSeek-R1-Zero says “Wait, wait.” mid-solution and reconsiders its first approach is an example. The small model on this page cannot show this.
Why it is still not training
| Post-training | Pseudo-learning (in-context learning) | |
|---|---|---|
| What changes | The weights | The context |
| How | Loss → backpropagation → correcting the weights, over many steps | One forward pass. Attention reads the context |
| How long it lasts | It stays in the next conversation | This conversation only. It is gone in a new one |
| With block 2 attention off | It remains: the L for |?C 1.000 → 0.758 | It disappears: all three contexts with examples answer J |
| Example on this page | |?C → L after 32 RL steps | |AKCNFJC → N right away |
In the experiment, questions with examples in the context were right 98.4% of the time; the same questions asked without examples in a new conversation, 10.2%. Meanwhile 1,500 inferences ran, yet the weights’ fingerprint stayed the same.
Why it looks like learning
In small Transformers, one linear self-attention layer could perform the same transformation as one step of gradient descent on the regression examples in its context, and attention-only Transformers trained on regression tasks actually did so. The explanation is that instead of correcting the weights, something resembling one step of training happens inside the computation that reads the context hypothesis. Whether large language models really do this is unknown.
That skill is made by training too. In the pre-training of this page’s model, in-context learning jumped from 33% at step 1,000 to 91% at step 3,000, while the one-block model never got past 36%. Pre-training puts the skill of following examples into the weights, and post-training (RL) grows the habit of reconsidering. The o1 announcement also puts both in one sentence: performance rises with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute).
When the two collide
Show a model post-trained on C→L and F→I a different mapping in its context, and the two compete. In |AKCPFJC it splits between the context’s P (0.602) and the taught L (0.349); in |AKCNFJF the taught I (0.980) beats the context’s J. The only pre-trained model follows the context: J (0.747).
Sources: Brown et al. 2020 abstract, §1 Figure 1.1 · DeepSeek-R1 paper §2.2.4, Table 3 · von Oswald et al. 2023 abstract · Olsson et al. 2022 (in-context learning score) · OpenAI, Learning to Reason with LLMs (2024-09-12, checked through secondary sources) · Small GPT: trained ourselves
SUMMARY
Do the weights change?
| Pre-training | Post-training | Inference (pseudo-learning) | |
|---|---|---|---|
| In English | pre-training | post-training | inference |
| Weights | change | change | stay as they are |
| When | Before release, first | Before release, after pre-training | After release, every time it is used |
| Who | The developer | The developer | The user |
| What it fits | The next token in the text | Example good answers (SFT), reward (RL) | Nothing. It makes an answer from the context |
| What remains | Language and knowledge | How to answer, how to work, the habit of thinking. It stays in the next conversation | This one answer. It is gone when the conversation ends |
Why it gets confusing
In Korean, one word, churon, covers both inference (running a trained model) and reasoning (working through several steps). The churon in “reasoning model” is reasoning, and that ability is made by post-training; the churon on this page is inference. One word, hakseup, likewise covers both training (correcting the weights) and learning in the broad sense (behaviour getting better). The learning in in-context learning is the broad sense. “Post” only names a time. Inference also happens after pre-training, so if you sort only by time, inference seems to fall under “post”. Put in time order, post-training comes before release and inference after it.
Common misconceptions
MythThe model learns during inference too.
ActuallyOnly inside the context. The weights stay as they are, so it is gone when the conversation ends.
MythPost-training is training after release.
ActuallyIt is training after pre-training and before release.
MythA reasoning model gets smarter while it is being used.
ActuallyThe habit of thinking longer arose during post-training. When used, it only spends more compute through that habit.
MythThe era of training is over.
ActuallyWhat was said to be ending is the path of scaling up pre-training. Post-training is counted as a scaling axis of its own.
Sources: Sutskever, NeurIPS 2024 talk (press reports) · TechCrunch 2025-01-07 (three scaling laws)
Confirmed · Hypothesis · Unknown
- ConfirmedIn its evaluation GPT-3 received tasks only as text, with no gradient updates and no fine-tuning. DeepSeek-R1-Zero applied RL to a base model with no SFT, and its answers grew longer. An induction head is a pair of attention heads in different layers and does not form in a one-layer model (papers).
- ConfirmedThe small GPT we trained: training changed the weights’ fingerprint and 1,500 inferences did not. In-context learning does not work with one block and works with two, and it disappears when block 2 attention is switched off. This is on synthetic text with 18 tokens and a 32-token context.
- HypothesisThat induction heads handle most in-context learning in large models too (indirect evidence). That block 2 of this page’s model gathers the positions that are not the answer and rules them out.
- UnknownHow in-context learning actually happens in large models. Whether post-training adds new knowledge or only selects behaviour that is already there.
Frequently asked questions
What is the difference between pre-training and post-training?
Both are training that sets a loss and changes the weights with backpropagation. Pre-training fits the next token in the text; post-training fits example good answers (SFT) or a reward (reinforcement learning). Post-training is done after pre-training and before release.
Does AI learn during inference?
The weights stay as they are. In-context learning, where the model follows examples put in the context, exists only within that conversation and is gone when it ends. The GPT-3 paper wrote that no gradient updates and no fine-tuning take place then (Brown et al. 2020).
What is in-context learning?
It is when a model follows a task it has never seen after a few examples are put in its context. It does not change the weights, so it is pseudo-learning that only looks like learning. The skill itself is made by pre-training (Brown et al. 2020 §1).
Is post-training done after release?
No. “Post” means after pre-training, and the developer does post-training before release. At release the weights are fixed, and what people use is inference.
Is the “reasoning” of a reasoning model the same as inference?
No. The reasoning of a reasoning model is working through several steps, and that ability is made by post-training (reinforcement learning). Inference is running a trained model to make an answer.
What did DeepSeek-R1 show about post-training?
DeepSeek-R1-Zero applied reinforcement learning straight to an only pre-trained model, with no SFT, and the rewards were rule-based scores for the right answer and the format. The longer it trained, the longer its answers became on average (DeepSeek-R1 paper §2.2).
Why does in-context learning need two layers?
The induction head, which continues [A][B] … [A] in the context with [B], is a pair of attention heads in different layers and does not form in a one-layer model (Olsson et al. 2022). In the small GPT we trained too, one block got the value of a key called again right 28–31% of the time and two blocks 98–99%.
What gets worse when post-training runs long?
Other skills can shrink. InstructGPT called the drop in performance on public NLP tests after RLHF an alignment tax, and reduced it by mixing in updates on the pre-training distribution (PPO-ptx) (Ouyang et al. 2022).
Further reading
- Backpropagation: pass the output’s error backward to correct the middle layer too
- Alec Radford et al., Improving Language Understanding by Generative Pre-Training (OpenAI, 2018)
- Tom B. Brown et al., Language Models are Few-Shot Learners (GPT-3, arXiv 2005.14165, 2020)
- Long Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT, arXiv 2203.02155, 2022)
- Ronald J. Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning (Machine Learning 8, 1992)
- Zhihong Shao et al., DeepSeekMath (GRPO, arXiv 2402.03300, 2024)
- DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv 2501.12948, 2025)
- Catherine Olsson et al., In-context Learning and Induction Heads (Transformer Circuits Thread, 2022)
- Nelson Elhage et al., A Mathematical Framework for Transformer Circuits (Transformer Circuits Thread, 2021)
- OpenAI, Learning to Reason with LLMs (2024-09-12)
- Computing, Sutskever warns AI will become less predictable (press report of the NeurIPS 2024 talk)
- TechCrunch, Nvidia CEO says his AI chips are improving faster than Moore's Law (2025-01-07)
- Experiment: the small GPT on this page (two blocks, 27,648 weights) was trained by us in Python (torch CPU). The comparison with one block is under “About this small GPT” in Try it.