KNOWLEDGE · OBJECT DETECTION
When one object gets several boxes, which one do you keep?
IOU (intersection area ÷ union area) measures how much two boxes overlap. Drop the low-confidence boxes first, then remove every box whose IOU with the most confident remaining box is above a threshold (NMS), and one box is left per object.
The YOLOv1 page uses IOU and NMS without explaining them. The two go together: IOU measures how much two boxes overlap, and NMS uses that value to keep just one of the boxes that overlap on an object. Try both in the two scenes below.
01 IOU
Intersection area ÷ union area
IOU (Intersection over Union) is the area where two boxes overlap divided by the area they cover together. It is 1 when they overlap completely and 0 when they don't touch. The PASCAL VOC object detection challenge counts a detection as correct only if its IOU with the ground-truth box is above 0.5.
Drag the predicted box to move it, and drag its bottom-right corner to resize it. With a keyboard, select the picture, move the box with the arrow keys, and resize it with Shift and the arrow keys.
IOU
0.46
Wrong by the VOC rule · not above 0.5
Intersection 8.4 ÷ union 18.2 = 0.46 (image area = 100)
It looks like a good overlap, but IOU is 0.46. By the VOC rule it is a wrong box.
IOU by box width, when a box is shifted sideways by 5% of the image width
Where YOLOv1 uses IOU
- Confidence: the confidence each box outputs learns its IOU with the ground-truth box. YOLOv1 01 Grid
- Responsibility: of a cell's two boxes, the one with the higher IOU with the ground truth takes the object.
- Penalty: the same size error lowers a small box's IOU more, so the network learns √w instead of w. YOLOv1 04 Training
- Inference: overlapping boxes on the same object are found by IOU and removed. 02 NMS below
- Scoring: the paper's score (mAP) counts a box as correct when its IOU with the ground truth is above 0.5.
Source: PASCAL VOC paper §4.2 Eq. 3, VOC2007 development kit §4.4, YOLOv1 paper §2, §2.2, §2.4, GIoU paper §3 · The IOU for a 5% shift is this page's calculation
02 NMS
One dog, one box
A network often outputs several boxes for a single object. In YOLOv1, several cells can pick up a large object or one that straddles cell borders. So the boxes are filtered twice to keep just one. ① Drop the boxes whose confidence is below a threshold. ② Of the remaining boxes, keep the most confident one and remove every box whose IOU with it is above a threshold; repeat ② until no boxes are left. Step ② is NMS (non-maximum suppression).
Click the step tabs in order to watch 6 boxes become 1. Then change the two thresholds or drag the boxes around. With a keyboard, use ← → on the tabs to move between steps; select the picture, pick a box with [ ], and move it with the arrow keys.
Done. Of the 6 candidates, 2 were dropped by confidence and 3 removed by NMS, leaving box A. One dog, one box.
The defaults, 0.2 and 0.4, are the values the YOLOv1 author's Darknet code uses. Lower the confidence threshold to 0.1 and box E, which covers only part of the dog, survives to the end: its overlap with box A is only 0.23, so NMS treats it as a box for a different object. Raise the overlap threshold to 0.7 and boxes B, C, and D stay too, so one dog ends up with four boxes. Set the overlap threshold too low, on the other hand, and NMS also removes the box of another dog standing close by, so that dog is missed.
Source: YOLOv1 paper Figure 1, §2.3, Darknet examples/yolo.c (the defaults in test_yolo), R-CNN paper §2.2, Soft-NMS paper abstract · The boxes are an example made for this explanation
Summary
From many boxes to one
- IOU measures how much two boxes overlap, from 0 to 1. A box whose IOU with the ground truth is above 0.5 is commonly counted as correct.
- Boxes whose confidence is below a threshold are dropped first. The YOLOv1 author's code uses 0.2.
- NMS keeps the most confident of the remaining boxes and removes every box whose IOU with it is above a threshold. Repeated until no boxes are left, it leaves one box per object.
MythAn IOU of 0.5 means the box is half right.
ActuallyShifting a same-size box by just ⅓ of its width, or only doubling its area, already brings IOU down to 0.5. PASCAL VOC set the bar deliberately low at 0.5 because the ground-truth boxes themselves can be inaccurate.
MythThe lower the NMS overlap threshold, the cleaner the result.
ActuallySet it too low and it also removes the boxes of other objects close by, so those objects are missed. Set it too high and one object keeps several boxes.
Source: PASCAL VOC paper §4.2, Darknet examples/yolo.c, R-CNN paper §2.2, Soft-NMS paper abstract and §4 · The IOU 0.5 examples are this page's calculation
Questions
Frequently asked questions
What is IOU (IoU)?
It is the area where two boxes overlap divided by the area they cover together (Intersection over Union). It ranges from 0 to 1: 1 when the boxes overlap completely and 0 when they don't touch. Also called the Jaccard index, it is used in object detection to measure how well a predicted box matches the ground-truth box.
How good is an IOU of 0.5?
Shifting a box of the same size sideways by one third of its width, or keeping its center and doubling its area, gives an IOU of 0.5. PASCAL VOC deliberately set the threshold as low as 0.5 because ground-truth boxes can themselves be inaccurate, and it counts a detection as correct only above 0.5.
What is NMS (non-maximum suppression)?
It is a post-processing step that keeps only one box when several overlapping boxes cover the same object. It keeps the most confident of the remaining boxes and removes the boxes of the same class whose IOU with it is above a threshold, repeating until no boxes are left, separately for each class.
Which comes first, the confidence threshold or NMS?
The YOLOv1 author's Darknet code first drops boxes with a confidence below 0.2, then runs NMS with an IOU threshold of 0.4 on the rest. Without the confidence filter first, a low-confidence box that covers only part of an object can slip through NMS and stay in the result. You can see this in 02 on this page by lowering the confidence threshold to 0.1.
How do you choose the NMS overlap threshold?
No value fits every case. A lower threshold also removes the boxes of other objects close by, so those objects are missed; a higher one leaves duplicate boxes. The YOLOv1 author's Darknet code uses 0.4, and according to the Soft-NMS paper, Faster R-CNN and R-FCN use 0.3 by default.
Where does YOLOv1 use IOU?
The confidence each box outputs learns its IOU with the ground-truth box, and of a cell's two boxes, the one with the higher IOU takes the object. At inference, NMS uses IOU to remove overlapping boxes on the same object. The paper's score (mAP) also counts a box as correct when its IOU with the ground truth is above 0.5.
Source: PASCAL VOC paper §4.2, VOC2007 development kit §4.4, GIoU paper §3, YOLOv1 paper §2, §2.2, §2.3, Darknet examples/yolo.c, R-CNN paper §2.2, Soft-NMS paper §4 · The IOU 0.5 examples and the results in 02 are this page's calculation
Further reading
- YOLOv1 Explained: Look Once, Find Every Object — where IOU and NMS are used
- Everingham et al., The PASCAL Visual Object Classes (VOC) Challenge (IJCV 2010)
- PASCAL VOC2007 development kit documentation
- Redmon et al., You Only Look Once (CVPR 2016) · Darknet examples/yolo.c
- Girshick et al., Rich feature hierarchies for accurate object detection (R-CNN, CVPR 2014)
- Bodla et al., Soft-NMS (ICCV 2017)
- Rezatofighi et al., Generalized Intersection over Union (CVPR 2019)
The scene and boxes in 02 are an example made for this explanation. Its IOU and NMS results were computed separately in Python and checked against this page's calculations.