Calibration is not mainly about whether one judgement happens to be correct. It asks whether expressed confidence matches outcomes over time. If events assigned an 80 per cent probability occur about 80 per cent of the time, those probabilities are well calibrated.
Calibration differs from accuracy. A system may answer correctly most of the time yet still claim 90 per cent confidence when it is wrong. Its accuracy may be respectable while its confidence is poorly calibrated. Conversely, assigning 60 per cent confidence cautiously does not by itself indicate weak judgement. Calibration describes the relationship between confidence and evidence, not the system’s entire ability to discriminate.
The concept links weather forecasting, medical judgement and AI output, and it also applies to personal thinking. Useful uncertainty is not vagueness; it keeps language from claiming more than the evidence supports. Recording predictions, confidence levels and later outcomes can reveal where we are routinely overconfident or overly cautious. Calibration does not weaken judgement. It teaches judgement to recognise its own boundary.
https://www.nist.gov/document/gorodnichy2dmitrycalibratedconfidencescoringforbiometricidpdf
https://scikit-learn.org/stable/modules/calibration.html
Discover more from Geoffrey Chen
Subscribe to get the latest posts sent to your email.