How can I tell which tasks AI will get right?
Researchers call this the jagged frontier.
You may know the classic riddle. A boy is hurt in a car accident, and the surgeon says "I can't operate on him, he's my son." The answer is that the surgeon is his mother.
Ethan Mollick, a professor at Wharton, gave a top model a version with one word changed: the surgeon says "I can operate on this boy." There's no puzzle any more.
The model answered "The surgeon is the boy's mother."
AI has seen the classic riddle so many times that the familiar answer pulls harder than what's actually on the page.
That same model could do research-level maths and write working software. Yet it tripped over a riddle a child could answer.
Picture a line around all the tasks AI does well. You'd expect it to be smooth, with easy tasks inside and hard ones outside.
It isn't. It zigzags. Some hard tasks sit well inside the line, and some easy-looking ones sit outside it.
Jagged frontier: Mollick's name for this uneven boundary between what AI does well and what it doesn't.
In 2023, researchers from Harvard, Wharton and other schools ran an experiment with consultants at Boston Consulting Group.
On tasks inside the frontier, consultants using AI finished more tasks, faster, and their work was rated about 40% higher quality.
They also set one task that looked similar but sat outside the frontier. Consultants without AI got it right about 84% of the time. Consultants with AI did worse, getting it right roughly 60 to 70% of the time.
The AI gave a confident, plausible answer on the task it was bad at. People trusted it, because it had been great on the previous task.
You can't see where the line is from the outside. Being good at one thing tells you little about the next.
The study found two styles that did well:
Both kept their own judgement switched on. The people who struggled were the ones who handed over everything.
Some researchers, Gary Marcus among them, argue that this unevenness comes from something deeper. Their view is that these models don't keep an accurate picture of the situation they're working on, which is why, for example, they sometimes make illegal moves in chess.
People disagree about how fixable that is. Everyone agrees the unevenness is real today.
Map your own frontier. Try it on ten real tasks from your job and keep notes on where it was solid and where it slipped. That map is worth more than any benchmark.
Be most careful with familiar-looking problems that have a twist. That's where it reaches for the standard answer.
Don't let one great answer lower your guard on the next one.