KNOWLEDGE · MACHINE LEARNING
Is train-val-test legacy?
No. calib is one more split for the values fitted with data after training ends: the temperature T, which corrects how large the confidence is, and the threshold, which turns a probability into a decision. The principle is the same as train-val-test: the data used to decide something cannot fairly measure that decision.
train fits the weights, val picks the hyperparameters and the model to use, and test decides nothing and only measures the score. Something is still left to decide after training: how far to trust the probabilities the model gives (the temperature T), and above what probability to call something a “defect” (the threshold). These two are also fitted to data, so depending on the data they are fitted on, the score inflates or collapses. The temperature scaling paper wrote that they may be fitted on the same val that picked the hyperparameters. calib is the split that takes over that job.
Temperature calibrates the values, and the threshold adjusts whether a value is adopted.
Sources: fitting temperature scaling on the validation set: Guo et al. 2017 §4 · calibrating on the training data biases it: Niculescu-Mizil and Caruana 2005 §2.1
01 Try it
Which split should fit T and the threshold
Here is a model that gets all 2,000 training examples right (synthetic data, 10 classes). Call class 0 a “defect” that must not be missed, and fit a threshold that catches 95% of defects. Change the split that fits T and the threshold. The score test reports and the real score on 27,000 examples seen for the first time after release change together.
| reported by test | real, after release | |
|---|---|---|
| confidence − accuracy0 means the confidence is honest | +0.6 pp | +0.3 pp |
| defect recalltarget 95% | 94.0% | 94.4% |
With T fitted on calib (2.62), the confidence matches the accuracy (+0.3 pp after release). The threshold fitted to exactly 95% on calib gives 94.0% on test and 94.4% after release. test is an honest estimate, so it moves up and down with every draw.
Each dot is one fresh draw of val, calib and test (100 draws). Across is the real recall after release, up is the recall test reports. On the dotted diagonal the report equals the real value; above it, test inflated its report.
Over 100 draws, test reports 94.5% on average and the real recall after release is 94.3% on average. test said 95% or more while the real recall was below 95% in 14 draws.
Numbers: computed by this page from synthetic data (10 classes whose true probabilities are known) and 12 small neural networks. Each draw uses the model with the best val accuracy. The threshold is the score that gives exactly 95% on the data it is fitted on (the ⌈0.95n⌉-th of n defects).
02 T
T only corrects how large the confidence is
A model is called calibrated when “90 of 100 predictions it calls 90% sure are right.” Modern neural networks are poorly calibrated, which sets them apart from those of ten years ago. On CIFAR-100 a 110-layer ResNet was far more confident than accurate, with an ECE (the average gap between confidence and the real share of right answers) of 16.53%.
Temperature scaling divides the logits z (the scores before the softmax) by T and then applies the softmax. T above 1 flattens the probabilities; T below 1 sharpens them. The logits (3, 1, 0) give 84 / 11 / 4% at T = 1 and 63 / 23 / 14% at T = 2. Every logit is divided by the same number, so the top class does not change. The accuracy stays the same and only how large the confidence is changes.
T is the single value that gives the smallest NLL (the average −log of the probability given to the right class) on calib. The more confident a wrong answer, the larger the NLL. The NLL is also why the confidence grows too large: even after every training example is right, more confidence still lowers the NLL on the training data, so long training leaves the accuracy where it is and grows only the confidence.
Each bar is the share actually right in that confidence bin (27,000 examples after release, 15 bins; bins with fewer than 20 predictions are not drawn). The closer the bars sit to the diagonal, the better the calibration.
mean confidence 93.5%accuracy 80.7%ECE 12.8%
At T = 1 this model is 93.5% confident on average but right only 80.7% of the time. At T = 2.62, fitted on calib, the mean confidence comes down to 80.9% and the ECE becomes 0.6%. Doing the same on train gives T = 0.05, because on the training data more confidence always means a lower NLL.
Sources: calibration, reliability diagrams, ECE (15 bins): Guo et al. 2017 §2 · temperature scaling and accuracy: §4.2 · overfitting the NLL: §3 · the ECE of ResNet 110: Table 1 · the numbers in the diagram: this page's synthetic experiment
03 Threshold
The threshold turns a probability into a decision
The model gives a probability; the threshold makes the decision, as in “raise an alarm if the probability of a defect is t or more.” The confidence threshold of 0.2 in YOLOv1, seen in IOU & NMS, is a threshold too. 0.5 is only a default. Move it yourself with the model from 01. Each dot is one calib example, and a dot to the right of the threshold raises an alarm.
defects caughtdefects missedfalse alarmsnormal, no alarm
The top row is the 106 defects in calib, the bottom row the 894 normal examples. Across is the defect probability, on a log scale to spread out the small probabilities near 0 (below 0.0001 at the left edge). The vertical line is the threshold, and t* is where the cost formula puts it. Drag the line or use the slider.
| calib (where it is fitted) | 27,000 after release | |
|---|---|---|
| defects caughtrecall | 101 / 106 (95.3%) | 94.4% |
| false alarmsshare of normal examples flagged | 72 (8.1%) | 8.6% |
| true among alarmsprecision | 58.4% | 54.5% |
| costmisses × 9 + false alarms | 117 | 127 per 1,000 |
threshold 0.0912: on calib it catches 101 of 106 defects (95.3%) and raises false alarms on 72 of 894 normal examples. After release it catches 94.4% of defects. With a miss 9 times as costly, the cost after release per 1,000 examples is 127 for this threshold, 126 for t* (0.100), and 127 for the cheapest threshold on calib (0.119).
Lowering the threshold misses fewer defects and raises more false alarms. Where it should sit is decided by the costs or by a target. If the probabilities are true, the threshold with the lowest expected cost is t* = the cost of a false alarm ÷ (the cost of a false alarm + the cost of a miss). Equal costs give 0.5; a miss nine times as costly gives 0.1. With a miss nine times as costly, t* = 0.1 costs 126 per 1,000 examples after release, far less than 0.5 (216) and about the same as the cheapest threshold on calib (127). This formula takes the size of the probability at face value. Press “not calibrated” and the same t* costs 141, more than the cheapest threshold on calib (128). That is why T is fitted first.
When the costs are hard to put into numbers, fit to a target. For “catch 95% of defects,” line up the defects in calib by score and read off the 95% point (exactly 95% on calib). For this model the threshold is 0.09. With 10 classes, the alarm has to go off once the defect probability passes just 9% to catch 95%.
On the data it was fitted on, it is always 95% or more. On new data it is not. If the threshold is the k-th highest score among n defects, a new defect lands above it with probability k/(n+1). With n = 100, exactly 95% (k = 95) gives 95/101 = 94.1%. Raising k by one, to ⌈(n+1) × 0.95⌉, brings the average above 95% (the +1 correction).
Even so, the result from one draw of calib wobbles. The wobble is set not by the size of calib but by the side the threshold applies to: the number of defects.
Recall on new data when calib is drawn afresh 2,000 times and the threshold is the ⌈(n+1) × 0.95⌉-th score. The bars span 5–95%, the vertical lines are the means (96.6%, 95.4%, 95.1%), and the dotted line is the 95% target.
With several classes, T can even change the order among examples. So the threshold is fitted on the probabilities after the T to be used is fixed.
Sources: costs and the threshold, 0.5: Elkan 2001 §1.3 Eq. 2, §2 · k/(n+1) and the +1 correction: Angelopoulos and Bates §1, Theorem 1 · the dots, the costs and the wobble: this page's synthetic experiment
Summary
Not a new rule, but the same principle used once more
- One decision per split. train fits the weights, val picks the model, calib fits T and the threshold, and test decides nothing. The number of splits is the number of steps fitted to data + 1.
- Where it was fitted changes the score. Fitted on train, it collapses; fitted on test, test inflates. Fitting it on the val that picked the model is usually fine, and in this page's experiment the difference was small too.
- Sometimes it needs its own split. Keep calib apart when a new instrument or site means refitting only T and the threshold, and when val was used to pick among very many candidates.
Mythtrain-val-calib-test is the new standard, so data must always be split four ways.
In factWithout values fitted to data after training, calib is not needed. With little data, T and the threshold can be fitted on val as well. The temperature scaling paper also fitted T on the validation set.
MythTemperature scaling raises the accuracy.
In factThe top class stays the same, so the accuracy is the same. What changes is how large the confidence is.
ConfirmedDefinitions, formulas, section numbers and quoted figures were checked against the originals. This page's numbers were computed from synthetic data and tested against the theory (k/(n+1), the beta distribution).
HypothesisWith real image models and real changes of instrument, the direction should be the same and only the size should differ. If val picked among very many candidates, T and the threshold should be biased as well (not seen in this experiment with 12 candidates).
UnknownNo fixed split ratio was found. The size of calib is set by the number of examples on the side the threshold applies to, and the size of test by the error allowed in the measure.
Sources: Guo et al. 2017 §4 · Niculescu-Mizil and Caruana 2005 §2.1 · Ovadia et al. 2019 §4.2
Frequently asked questions
Is the train-val-test split legacy now?
No. train fits the weights, val the hyperparameters and the model, and test decides nothing and only measures the score. calib is one more split, kept when the temperature T and a threshold are fitted with data after training. The temperature scaling paper (Guo et al. 2017 §4) wrote that calibration may use the validation set that picked the hyperparameters.
How is a calibration set different from a validation set?
The validation set picks the model and the hyperparameters; the calibration set fixes how to read the chosen model's output (the temperature T, the threshold, the q̂ of conformal prediction). With little data, both are sometimes done on the validation set (Guo et al. 2017 §4, Niculescu-Mizil and Caruana 2005 §2.1). Calibrating on the training data biases it (the same §2.1).
What is temperature scaling?
A calibration that divides the logits by a temperature T before the softmax. T above 1 flattens the probabilities and lowers the confidence; T below 1 sharpens them. T is the single value that gives the smallest NLL (cross-entropy) on the calibration set (Guo et al. 2017 §4.2).
Does temperature scaling improve accuracy?
No. Every logit is divided by the same positive number, so the top class does not change and the accuracy stays the same (Guo et al. 2017 §4.2). What changes is how large the confidence is. A 110-layer ResNet on CIFAR-100 went from an ECE of 16.53% to 1.26% (Table 1 of the same paper).
Why not leave the classification threshold at 0.5?
0.5 is right only when both kinds of error cost the same and the probabilities are true. Fitted to costs, the threshold is the cost of a false alarm ÷ (the cost of a false alarm + the cost of a miss), which is 0.1 when a miss costs nine times as much (Elkan 2001 Eq. 2). For a target such as “catch 95% of defects,” read that point off the scores in the calibration set, and measure the result separately on test.
Does conformal prediction use a calibration set too?
Yes. Take the ⌈(n+1)(1−α)⌉-th of the scores 1 − (probability of the right class) over n calibration examples as the threshold and give every class past it as a candidate answer: on new data from the same distribution, the right answer is among the candidates with probability at least 1 − α. With 1,000 calibration examples the coverage is usually between 0.88 and 0.92 (Angelopoulos and Bates §1, §3.2).
Does calibration need to be redone when the instrument or the data changes?
Yes. Calibration on an i.i.d. validation set does not guarantee calibration on data whose distribution has shifted (Ovadia et al. 2019 §4.2). The model can stay as it is while a new calibration set from the new setting refits T and the threshold. In this page's synthetic experiment, on data with 1.3 times the noise the old values gave a recall of 83.8%, and refitting on 1,000 new examples brought it to 95.4%.
Sources: Guo et al. 2017 (§4, §4.2, Table 1) · Niculescu-Mizil and Caruana 2005 §2.1 · Elkan 2001 Eq. 2 · Angelopoulos and Bates (§1, §3.2) · Ovadia et al. 2019 §4.2
Further reading
- Guo, Pleiss, Sun, Weinberger, On Calibration of Modern Neural Networks (ICML 2017)
- Angelopoulos, Bates, A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
- Niculescu-Mizil, Caruana, Predicting Good Probabilities With Supervised Learning (ICML 2005)
- Elkan, The Foundations of Cost-Sensitive Learning (IJCAI 2001)
- Ovadia et al., Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (NeurIPS 2019)
This page's numbers were computed from synthetic data (10 classes, 30-dimensional normal distributions) and 12 neural networks with two hidden layers. The same computation was tested against the theory. The numbers for real image models may differ.