Why a 95% Accurate Model Can Still Fail Unevenly
A model that's 95% accurate overall can still be far less accurate for one group than another, usually because its training data didn't look like everyone it ends up treating. Compare subgroup accuracy below and see how easily that gap hides behind one clean overall number.
Worth reading first: A Second Opinion, Not a Replacement
“95% accurate” sounds like a single, settled fact about a model. It almost never is. A model that is 95% accurate overall can still be far less accurate for one group of patients than another, and the one clean number on the slide is usually the reason nobody in the room noticed.
95% accurate overall can still fail unevenly
Overall accuracy is an average across every case a model was tested on. Averages hide exactly the kind of unevenness that matters here — a model can be excellent for most patients and meaningfully worse for a specific group, and the overall number will still look reassuring, because the group doing well outnumbers the group doing badly.
Training data that doesn't look like everyone it treats
The chart below simulates exactly this. Same model, same day, one overall accuracy figure — and three subgroups underneath it that received wildly different shares of the training data before the model was ever deployed.
The headline number — 91.0% accurate overall — is real. It is also an average that hides a 16-point gap between the best-served and worst-served group, entirely explained by how much of each group's data the model actually trained on.
Why checking subgroup accuracy isn't optional
A group that made up 7% of the training data is not a rounding error to that group — it is the entire population the model performs worse for, every single time it is used on someone from that group. The model was never lying about its overall accuracy. It was just never asked to report the number that actually mattered to the patients it was failing.
Key takeaways
- Overall accuracy is an average, and averages can hide a large gap between a model's best-served and worst-served groups.
- A subgroup that made up a small share of the training data can experience meaningfully worse accuracy, invisible in the one overall number.
- The disparity is usually not deliberate — it traces back to which patients were, and weren't, well represented in the training data.
- Checking subgroup accuracy before deployment is not extra diligence — it's the minimum bar for claiming a model is safe for everyone it treats.