One good AI answer doesn't prove it works

"It worked when I tried it" isn't the same as "it works".

The demo that went well

You write a prompt that pulls the totals out of supplier invoices. You test it on one invoice. It's perfect.

You roll it out to the team. A week later, finance finds three invoices where it read the VAT line as the total.

The prompt didn't get worse. You only ever saw it work once.

Made-up example.

Each answer is one roll of the dice

Ask it the same thing twice and you can get two different answers (see "Same question, different answer").

So when it gets something right, you've seen one possible answer. You haven't seen how often it gets it right.

Nielsen Norman Group put it simply: one AI output is an example, not an evaluation.

Evaluation: checking how well something works across many tries, instead of one.

Works once vs works every time

These are two very different claims.

• "It can do it": it got it right at least once in a few tries. Fine for things you'll check yourself, like a first draft.
• "It does it reliably": it gets it right nearly every time. That's what you need for anything you won't check, like a process the whole team relies on.

A lot of disappointment comes from mixing these up.

How the numbers add up

Being right 90% of the time sounds great. Spread across a month of invoices, it's a lot of mistakes.

Your tries weren't the hard ones

When we test something, we tend to try the easy, typical case first: the clean invoice, the short email, the simple question.

Real work is messier. The scanned invoice is crooked, the email has a forwarded thread underneath, the question uses your company's jargon.

If you haven't tried the awkward cases, you don't know yet.

Same for other people's demos

A slick demo video, a colleague's "look what it did", a launch post with one stunning example: these are all single tries, usually the best one.

They show what's possible. They don't show what's typical.

Try this

Run it more than once. Three to five tries on the same input shows you how much the answers vary.

Try the awkward cases on purpose: the messy file, the long thread, the edge case you're worried about.

Before you rely on it, build a small test (see "Test AI on five real examples").

Sources

• Nielsen Norman Group, "Evaluating AI output": nngroup.com/articles/eval-ai-output/
• Hamel Husain and Shreya Shankar, "Evals FAQ": hamel.dev/blog/posts/evals-faq/