Survivor bias: we only hear from the people who made it
Updated: Oct 27
"I made it, so anyone can".
It's one of the most sincere things people say about success. Someone who started with little, worked hard and built a good life looks back and draws what seems like the obvious lesson: the path is open to anyone willing to walk it.
But how would they know? Their success story doesn't show how many people tried the same path and didn't make it.
This is known as survivor bias.
What survivor bias is
Survivor bias happens when we draw conclusions from the people or things that make it through a filter. A famous example comes from World War II. The story goes that the military studied bombers returning from missions and planned to add armour where the planes had the most bullet holes. However, statistician Abraham Wald pointed out that they were only looking at the planes that survived. The armour actually belonged on the unhit areas, because the planes that were shot there never made it back to be studied.
Meritocracy is survivor bias at scale
"I made it, so anyone can" is meritocracy in a single sentence.
It is the belief that success reliably goes to the people with the most talent and effort, because everyone has the same chance to get there.
Don't get me wrong, succeeding on merit is a good thing. People can and do succeed through talent and hard work.
But merit is about individuals, and meritocracy is a claim about a whole system.
Meritocracy relies on everyone starting from the same place, and they don't. Some people start with family support, well-resourced schools, useful connections or a safety net if things go wrong. Others start without any of these things.
Sociologist Michael Young coined the word as satire. He imagined a future where the people sorted to the top came to believe they deserved their position.
Despite the word being meant as a warning, many politicians have adopted meritocracy as an ideal. In Aotearoa, a new law reinforcing "merit-based" appointments in the public service shows how live this idea still is, and the US is reshaping federal hiring and contracting around merit.
Survivor bias is how stories of merit turn into a belief in meritocracy. Someone's success can be fully earned, and still tell us nothing about the people who worked just as hard and didn't make it.
The survivor filter
As of October 2024, 9 of 122 NZX-listed companies had a woman as chief executive. The usual explanation for this gender disparity is the pipeline: there aren't enough women with the right experience. To be considered for chief executive, you first need a senior role that leads to the top job. To get one of these, you need to have made it through every promotion before it. With each step up, only the people who survived the last one are considered.
Here, the pipeline is the filter.
Seen through the lens of meritocracy, the process looks fair, because each step judged people on individual merit. That's fair to the people still in the running. But it can't account for the reasons others aren't, such as bias at earlier stages, caring responsibilities or jobs designed around long, inflexible hours. To judge the system, you need to see who started out and where they dropped away. Saying there aren't enough women with the right experience describes what the filter produced, without explaining why.
Surviving the filter brings power
The people who survive the filter make up the dataset — like the chief executives of NZX-listed companies. They're also the ones who read the data, set the criteria and decide who comes next.
Survivor bias gives them an easy way to see the system as fair. They worked hard and succeeded, so the filter they made it through can seem fair too. Their stories can also make success seem equally within reach for everyone, and make support for others seem unnecessary.
None of this needs bad intent. A leadership team reviewing its promotion data is looking at real people who really did well. The people filtered out earlier simply aren't in the data.
Survivor bias in disparity data
NCEA is the main qualification students work towards in their final years of secondary school in Aotearoa. Each year, the government reports what percentage of students achieved it.
For years, this was calculated using only the students on the school roll as at 1 July. Anyone who had already left school that year was filtered out. But students who leave partway through the year are likely to leave without completing the qualification. So excluding them increased the proportion who did. In 2017, about 4,000 of the students filtered out were Māori.
From 2019, this metric was changed to count every student enrolled during the year. When the government recalculated the 2017 results using the new measure, nothing about the students or their results had changed, only the way they were counted. The percentage of Māori students gaining NCEA Level 2 fell from 74.4% to 70.4%. For European students, it fell from 84.5% to 82.0%. The gap between these groups grew from 10.1 to 11.6 percentage points. At NCEA Level 3, the gap grew even more, from 13.7 to 16.3 percentage points.
Changing who was counted changed the size of the disparity.
Students gaining NCEA Level 2 in 2017 (%)
What to do about it
Before you assume a dataset or a success story applies to everyone, find out who's missing from it.
Look for who was filtered out. Many datasets filter people for reasons that seem sensible, like only counting people still enrolled on a certain date, or only including people who finished a programme or answered a survey. If you have data on the people who were filtered out, add them back in and run the numbers again, as in the NCEA example. If you don't have their results, compare what you do know about them, such as age, ethnicity or region, with the people who were included. If the two groups look different, your results probably don't reflect everyone in the starting group.
Report every stage. A final number, like the percentage of women chief executives, hides where people were removed from this pipeline. So, measure this percentage at each step along the way, from entry-level positions to the top job. The step where the number drops most is your sign to start looking for what's excluding people.
Ask for the starting number. When a success story is used as evidence that a system works, ask how many people started out on the same path. Without that, one success tells you very little about everyone else.
The next time you hear "I made it, so anyone can", ask who else tried.
I write about data and power, so if you'd like to follow along, the best place to do that is my newsletter. This article is part of the Things We Tell Ourselves series. Each article looks at an everyday saying, what the data shows, and what it means if you tell stories with data.
Kat Greenbrook is a data storytelling consultant, author, and workshop facilitator based in Wellington, New Zealand. She is the founder of Rogue Penguin and the author of The Data Storyteller's Handbook.
