Thoughts on Survivor Bias

Over the course of the past years I‘ve become more and more interested in cognitive biases, how they affect our work and what could be done to mitigate them. A couple of months ago I‘ve published my thoughts on groupthink. Today I want to share some thoughts about survivor bias. I believe it‘s hard to exaggerate the effect of this bias, but at the same time it seems to be relatively unknown in our community.

Never give up!
For whatever reason Facebook decides to show me these "motivational" pictures with "deep" messages every now and then. Yesterday I saw this gem:

Seems like good advice. It must be good advice to never give up, if this was the secret sauce for Coca-Cola‘s success, right? Of course it‘s not! It‘s pure survivor bias. Just as well you could ask a millionaire what her recipe for success was and then she would reply: "Quit your job, sell your house and your car, go to the casino and bet everything on number 12! That‘s exactly how I got rich." Of course that‘s bad advice, but unfortunately this is exactly how many business books and other guidebooks work.
We‘ll come back to this later, but first let‘s hear about a story from World War II.

Abraham Wald
The Hungarian statistician Abraham Wald was part of a group of geniuses, who helped the US win World War II with weapons of mathematics. The group was called "Applied Mathematics Panel", and there‘s a wonderful article on this group, written by David McRaney. One task the group was assigned to was to figure out how to improve the armor of allied bombers. It was clear they needed extra protection (too many were destroyed in combat), but it was unclear were exactly to put the additional armor, and protecting the whole plane would make it too heavy. So commanders examined the planes that returned from battle and looked at the bullet holes. It was easy to spot patterns (one obvious one was along the wings), so they concluded to add extra armor at exactly these places. Fortunately, Wald was involved in solving this problem, and his clear thinking probably saved hundreds of lives. He concluded, that the exact opposite of this had to be done: improve the armor at the places where no bullet holes could be spotted. His reasoning went like this: The planes we get to see here all have survived the battle, so they have done pretty okay. But all the planes who did not return from combat would have needed extra protection, because they had been hit at all the places we don‘t see at our planes here.

Survivor Bias
Wald‘s story perfectly explains what survivor bias is:

The logical error of concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility. (Wikipedia)

It would be easy to smile at this story from World War II and just carry on. But survivor bias has serious implications for almost all areas of our professional and personal lifes. Here are some examples:

• Ever heard someone say this:
  "The quality of XY nowadays is so crappy. I have a really great 20-year old XY, but they don‘t make these anymore"?
  I‘ve certainly said something like this many times. However, chances are that this statement is not true. In fact, the quality of many things has drastically improved over the last decades. They‘ve made a lot of crappy items in the past as well, but they all have disappeared, so that only the (few) good ones have survived. Or, if you look at it the other way around: They build good ones and crappy ones today, but we haven‘t figured out yet, which ones are the good ones.
• We tend to believe that the cities of eg the Renaissance must have been really beautiful, and all the art of this time was extraordinary. This must be the case, because all the buildings and all the art from the Renaissance we get to see is very beautiful. But of cause we never get to see all the ugly buildings, paintings, songs etc., because they were torn down, cast away, not passed on and not talked about in history books.
• Social media is a big survivor bias machine. Whenever I browse through my Facebook stream, I get the impression that all my friends have spectacular lifes, while mine is nothing but boring. Wherever I look: Happy people with a drink in their hands (one with an umbrella!), travelling the world, chilling at the beach in wonderful weather. At least two survivor biases at work here: 1) People only take pictures when the skies are blue and the wine is plenty. 2) People tend to only post spectacular stuff. They rarely post pictures of them being bored on a cloudy day.
  In the case of social media, events "survive" the processes of photography and social media posting only if they are interesting and colorful.
• I do not understand a lot about mutual funds, but I stumbled on this paper, whose authors claim that "[a]lmost all prior mutual funds studies suffer from survivorship bias".
• Ever heard of this anecdote? When cats fall from a balcony, they have a bigger chance to survive, when they fall from a greater hight! The data for this conclusion was drawn from the observation that cats that were brought to the vet had greater injuries if they had fallen from 1-6 stories, compared to cats who had fallen from higher than six stories. One explanation given was that whenever cats fall from a greater hight, they have enough time to spread their legs and use their body as some kind of parachute. Sounds amazing, but perhaps the more reasonable explanation for this phenomenon is this: Cats who fall from higher than six stories are rarely taken to the vet, because they did not survive the crash or, if they do, their owners don‘t find them or do not have any hope for cure.

These are just some examples from very different areas, and the list could go on and on. What all these stories have in common is that the conclusions we draw from the data are backward looking: We look at a set of visible data and conclude what must have happened in the past. That seems reasonable, but as Gary Smith claims in his book Standard Deviations: "When we chose a sample today and look backwards, we only see the survivors" and that those studies often suffer from survivor bias. From a scientific standpoint we should have done the following to conduct serious, forward looking research: Take 100 random cats, for each one toss a coin, throw half of them from the 3rd story and the other half from the 8th story and then see which group has the greater injuries. No, I am in no way proposing actually doing this, but you get my point:-)

Great Companies
Now what has all this to do with the Lean/Kanban community? I think one serious implication is best illustrated by Jim Collins‘ best-selling book "Good to Great". Collins compares eleven "great" companies and then finds a set of seven characteristics all these companies share. His conclusion is that he has found the recipe for success and if you use this recipe, your business will be successful as well. Gary Smith points out that Collins‘ approach is pure survivor bias, because it‘s backward looking. Collins only looks at companies that still existed at the time of the writing of his book. So we have no way of telling how many other companies did the exact same thing but went bankrupt anyway. Smith also points out that the eleven "great" companies do not perform that great anymore: Between 2001 and 2012, these companies combined did worse than the overall stock market.

Now if even Collins‘s work (who worked as a Stanford professor) is affected by survivor bias, we can be sure that many, many other articles and books are as well. So we should be very careful, whenever we read something about "the 10 things that all successful managers/teams/organizations do" or the like. Plenty of these writings exist in our community, as well. I am not suggesting that these stories are useless. I find many inspiring ideas in them, many worth trying and adapting. I just disagree with the (implicit or explicit) conclusion that we, too will become successful by doing the same thing.

P.S. I‘ve also written a blog post on confirmation bias you might also enjoy reading.