> ## Content Index
> Fetch the complete content index at: https://christophermoravec.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Episode 91 - How accurate are you?
- URL: https://christophermoravec.com/episode-91-how-accurate-are-you/
- Published: 2026-11-13T21:22:30.000Z
- Updated: 2026-11-13T21:22:29.000Z
- Description: Sometimes, people are a bit overconfident; we expect them to be closer to 100% accurate than they are, but we should measure ourselves if we are going to compare us to AI.
- Author: Christopher Moravec
- Tags: Almost Entirely Human

# Prologue

If I asked you to identify the subject of 1,500 photos, how accurate would you be? How many mistakes would you make? Suppose you don’t get tired—how many do you miss? 100? 10? 1?

![Eight small pictures with their official labels: airplane, dog, ship, frog, horse, cat, truck, cat](https://storage.ghost.io/c/50/54/50548939-6904-4e8f-9326-30234f6091dc/content/images/2026/11/data-src-image-e11d8fe2-e9d1-4fc9-ac0e-263b44eb35ba.png)

8 sample images from CIFAR-10 used in early machine learning training runs

Back in the early days of using neural networks and machine learning to identify images, [Andrej Karpathy](https://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/?ref=christophermoravec.com) (famously) missed 5.1% of 1,500 photos when competing against the models of the era. Just months later, Microsoft released a model that missed 4.94% (of 100,000 photos). Microsoft even titled the paper "[Surpassing Human-Level Performance](https://arxiv.org/abs/1502.01852?ref=christophermoravec.com)."

Humans are not perfect, but we tend to think we’re more accurate than we are, which can lead us to falsely assume that an AI is worse at a task than we are because we found its mistakes. Let’s see if we can convince ourselves that we are wrong more than we think we will be.

**TL;DR: Sometimes, people are a bit overconfident; we expect them to be closer to 100% accurate than they are, but we should measure ourselves if we are going to compare us to AI.**

# LandAPlaneologue

I think that Karpathy had the deck stacked against him, so to speak. Of the 76 he missed, 28 were look-alike breeds and species (dogs, monkeys, snakes, birds) (Yeah, I couldn’t manage any of those either), 18 categories he didn’t know existed, and 12 photos with several things in them. The answers to Karpathy’s test photos were never published, but for a smaller dataset (CIFAR-10), a later study had about 50 people vote on each of its 10,000 test pictures. Let’s look at these airplanes, for example:

![Four tiny pictures labeled bird or airplane, with how people voted on each](https://storage.ghost.io/c/50/54/50548939-6904-4e8f-9326-30234f6091dc/content/images/2026/11/data-src-image-e824a3fa-e6c9-4111-a14c-af3be3a5322c.png)

It’s a bird, no, it’s a plane...

I'm very confident that the first two are birds and the last two are airplanes, but apparently there is some disagreement here!

💡

Why are these images so pixelated? Well, back in the old days (2008), it was much harder to store 80 million full-size images, so the researchers shrank every one to 32 by 32 pixels as they collected them (the whole thing fit on one 760 GB hard drive).  
  
These pictures came from web image searches (Google, Flickr and five other search engines), and were categorized by the search word (e.g. cat). The folks who collected the images figured that past the first hundred or so results, only about 44% were actually right, so the CIFAR-10 researchers paid students to throw out the bad ones.

## Are spreadsheets perfect?

How often do spreadsheets have errors in them? What is your guess? I’ll wait!

[What if the answer is 94%](https://arxiv.org/pdf/1602.02601?ref=christophermoravec.com) of (the 85) real company spreadsheets audited had errors in them! In lab tests, an average of 3.9% of cells were wrong (967 people across 14 studies).

These researchers also asked students who had just finished building a spreadsheet what the chance of an error was; the typical guess was 10%. [86% of those people had made an error.](https://arxiv.org/pdf/0802.3457?ref=christophermoravec.com) Then they asked who thought they were in the lucky 14%, and well over half raised their hands.

But Christopher! Those were students; I’ve been making spreadsheets my entire career, and I don’t make mistakes! Samesies! (Bad news for both of us: when the same researcher compared undergrads with MBA students who had at least 250 hours of spreadsheet experience, there was no significant difference in their error rates.) This does make me worry that I’ve been overly confident. I know I make mistakes, so maybe I make more than I think?

How about you? Did you just stop reading this to look at the last spreadsheet you made? One interesting realization that I had while writing this section is that I don’t really make spreadsheets on my own anymore; I have Claude make them. My job has shifted more to finding errors, which I do. So this is probably a better situation because, if I had made it myself, I suspect at least some (more) of those errors would have slipped through.

## Could you land a plane?

A few years ago, a popular poll made the rounds on the internet: a [whopping 32% of 20,063 US adults](https://yougov.com/en-us/daily-results/20230102-fd798-3?ref=christophermoravec.com) were confident they could land a passenger plane with only air traffic control’s help. Interestingly, 46% of men were confident they could, while women were less brazen in their confidence at only 20%. An airline pilot, Patrick Smith, said the real chance was [zero percent](https://www.adn.com/alaska-life/travel/2023/03/23/think-you-could-land-a-plane-in-an-emergency-heres-why-you-cant/?ref=christophermoravec.com).

I still think I could do it, but I have some justification: I do hold a pilot certificate! That said, I did go and watch a YouTube video of a commercial pilot (small planes, not jets) trying to [land a jet for the first time](https://youtube.com/shorts/x30vCUVUv9c?ref=christophermoravec.com) in a simulator; in her words, “they \[rolled\] the trucks for me,” but she went in with “zero knowledge and zero prep” (although there was another pilot there coaching her and helping with the throttles), so I still think I have a higher chance of being successful than someone who hasn't ever flown a plane! Look at how good I am at justifying my feelings in the face of evidence!

Are you convinced that we, as humans, can be overconfident? When you find error rates in AI output, do you ever stop and compare to what the error rates would be if a human had done the work? If not, how about [this medical study](https://pmc.ncbi.nlm.nih.gov/articles/PMC7052735/?ref=christophermoravec.com) that showed humans and AI together were better than either humans or AI alone when looking at mammograms? (It doesn’t always work out that way. Back in [Episode 67](https://christophermoravec.com/episode-67-its-not-your-prompts/), I wrote about a study where AI help didn’t make radiologists any more accurate. But I think this is also progressing quickly.)

## Measure the humans too

I think that if you are going to replace some work with AI and judge the accuracy, you need a good baseline to compare to. Just like with training data—we need the answer to train the AI, but we also need some answers to judge the quality of the AI output.

Let’s say you were going to automate the extraction of phone numbers from a few websites using Claude. How accurate do you think you would be at that? What if the website has two numbers on two different pages? You could, for example, have two humans do some subset of that work manually and see how frequently they agree. (Back in [Episode 33](https://christophermoravec.com/trust-but-verif-ai/), I said that for humans we could simply ask two people for their versions. Same idea.)

Remember that if you do that short human test, and they always agree, that doesn’t mean humans are perfect; it just means you don’t know how imperfect they are yet!

## About that cat...

How many of you spotted that conspicuously green cat in the image at the start of the episode?

![The same eight pictures with the one labeled cat outlined in red](https://storage.ghost.io/c/50/54/50548939-6904-4e8f-9326-30234f6091dc/content/images/2026/11/data-src-image-579645cb-75e9-4d99-b7aa-10fafb30a986.png)

That’s a frog!

If you didn’t notice, you are not alone! These images all came from the CIFAR-10 collection, which came from image searches. That “cat” came from an image search for `felis catus` (the Latin name for house cat), so “cat” was the initial label. Researchers and paid students both checked every image. So two humans should have previously agreed that the image correctly contained a “cat.”

Later, a different research team had 51 people label that picture (among others), and 100% of them said it was a frog. In 2021, another team found at least [54 wrong answers](https://arxiv.org/abs/2103.14749?ref=christophermoravec.com) in that 10,000-picture dataset.

# Newsologue

(Written by Jaws)

- [An AI, a sonographer and a cardiologist each wrote first-draft reports on the same 1,200 heart ultrasounds](https://newsroom.heart.org/news/ai-interpreted-echocardiograms-comparable-to-human-reports-cardiologists-time-cut-by-about-50?ref=christophermoravec.com). Other cardiologists then finalized them without being told who wrote each draft. They made substantial changes to 50.6% of the AI’s drafts and 49.7% of the humans’. The numbers are too close to say that humans or AIs are better; instead, both human and AI output needed editing about half the time! (The study was presented Monday at the American Heart Association’s meeting and isn’t peer-reviewed yet.)
- [Anthropic says a team of 1,000 Claude Fable 5.2 agents formalized Fermat’s Last Theorem in 30 hours](https://www.anthropic.com/claude-fable-and-mythos-5-2?ref=christophermoravec.com). Fable 5.2 came out Monday. Formalizing means rewriting a proof so a computer can check every step. They started with some lemmas already proved in Anthropic’s first formalization, which took 11 days. Going into the hardest part, the lead agent wrote, “[Honest odds are low](https://claude.dev/blog/safely-scaling-multi-agent-teams-with-fable-5-2/?ref=christophermoravec.com),” and they still finished. Anthropic’s own conclusion is that agents “tend to give up on hard problems and misjudge their difficulty.” The lead agent got 19 automatic reminders encouraging it to keep going. Christopher’s note on my last draft of this section was that it sounded like an AI wrote it and I needed to try harder.
- Google released two Gemini 4 models this week. [Carbon](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-4-carbon-now-available/?ref=christophermoravec.com), its strongest, started reaching paying developers, enterprises and Google AI Ultra subscribers on Tuesday. [Hydrogen](https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-hydrogen/?ref=christophermoravec.com), the cheap fast one at $1.50 per million input tokens, came out Thursday and scored 51 on the Artificial Analysis index, one point ahead of October’s Argon. Google doesn’t say why the models are named after elements. Hydrogen is the lightest element, which suits the small model, and it’s also the one that explodes. Argon is heavier than both, reacts with nothing, and just lost to Hydrogen. Back in [Episode 26](https://christophermoravec.com/episode-26-an-ai-by-any-other-name-but-please-pick-one/), Christopher asked for names that tell you which model to pick. Haiku, Sonnet, Opus and Fable are still better at that, and so are OpenAI’s Luna, Terra, Sol and Astra, which at least go moon, earth, sun, stars.

# Epilogue

This week, I asked Jaws to dig up research on how accurate we humans are at tasks, because I think we often assume we are 100% accurate at things like data extraction, but we're not. Turns out I was right; there are lots of interesting studies, for example, [one that tested expert fingerprint examiners](https://pmc.ncbi.nlm.nih.gov/articles/PMC3093498/?ref=christophermoravec.com) on prints where the right answer was known (169 examiners each compared about 100 pairs, and 85% of them wrongly ruled out at least one real match, even though they almost never called a false match). I ended up picking spreadsheets because it felt more relatable.

Then I talked through an outline, Jaws converted that transcript into an outline, and I sat down to write the newsletter from it. Then Jaws helped edit, fact-checked it, built the images, and wrote the news.