When should I expect AI to be wrong?
Calibration: knowing when to expect a wrong answer.
A good forecaster who says "70% chance of rain" is right about 70% of the time when they say it. Over a year, 7 out of every 10 "70% days" are wet.
That forecaster is calibrated.
Calibration: how well someone's confidence matches how often they're right. If you say "90% sure" and you're right 9 times in 10, you're well calibrated.
A model's confidence and its accuracy don't track each other closely.
AI sounds the same when it's right as when it's wrong (see "AI isn't looking anything up"). And when you ask it how sure it is, the number it gives you is more generated text. It isn't a measurement.
Anthropic's course on AI limitations lists "loose confidence calibration" as one of the habits these models pick up in training.
When OpenAI published its GPT-4 report in 2023, it showed something interesting.
Before the final round of tuning, the model's confidence lined up quite well with how often it was right. After the tuning that made it a helpful assistant, the match got noticeably worse.
The tuning rewarded answers people liked, and people like confident answers.
The second one is the one that matters day to day.
The Calibration Game shows you 20 questions people asked an AI. For each one, you guess how likely it is that the AI got it right, and at the end you get a score for how well calibrated your guesses were.
Most people are surprised by which questions it fails on.
People who use AI a lot tend to learn the warning signs:
Over time you get a feel for when to double-check. This lesson is meant to speed that up.
Ask where it's weakest, not how sure it is. "Which part of this answer is most likely to be wrong?" That question tends to be more useful than a percentage.
Make a prediction before you check. Before you verify something, guess whether it'll turn out right. Keep score for a week and you'll calibrate quickly.
Ask twice. If two fresh chats give different answers, treat both with suspicion.