My headline is exactly the sort of thing that Dale Carnegie warned me not to say if I want to make friends. But, I already have plenty of friends, and occasionally I like to throw a cat among the pigeons.
So, before you start to comment your vehement disagreements while tagging your preferred business influencer, hear me out. You can always unfollow me later.
I’m referring to that guy (and they are overwhelmingly male) who emphatically shares his Secret Success Recipes™ online, in books, on podcasts, and especially on those snackable* videos littering your social feed. This guy confidently proclaims, with raised voice and a cockiness-level that would make Muhammad Ali blush, that he knows exactly how you can grow your business and get stupidly rich, just like him.
“Just do what I did.” he shouts at you, pointing at the webcam or looking down at you from the stage. He’ll even take a minute to insult you for being lazy or scared, just because he tells it like it is. Hordes of adoring fans lap up everything he has to say (and sell), madly retweeting and liking and sharing until their thumbs ache.
Sure, he occasionally embraces moments of humility, explaining that he’s made plenty of stupid mistakes along the way and that he’s really a down-to-earth guy, just like you. The fancy car / private jet / waterfront mansion which feature in his promo videos are great, but underneath all that he’s just a blue-collar kid that hustled his way to the Big Time.
Here’s the thing: that biopic may be absolutely correct. He may be down-to-earth and self-made. He may have hustled his way to millions and still love the grind. He may even genuinely care about you and all his followers and believe his own rhetoric. But none of this matters, because his advice is lousy.
It’s lousy because his passion and zeal glosses over the lack of evidence for his assertions and distracts you from the heaping spoonful of bias that he’s asking you to swallow. This wouldn’t be an issue if he was a sportscaster or gossip columnist, but instead he’s asking you to gamble with your business, your livelihood, your future.
At the heart of the problem is a phenomena known as survivorship bias. This error occurs when we look at an outcome, and presume that certain elements preceding the outcome are the cause.
The most often cited example for this relates to a statistician who exposed the error during World War II. Abraham Wald disagreed with the navy, who looked at damaged aircraft returning from missions and who recommended that better armor needed to be added where the shrapnel had left damage.
Wald disagreed, rightly pointing out that those planes that never made it back had probably taken hits around the cockpit, engines, and fuselage. The returned planes that the navy were looking at were the survivors. The armor needed to be added to the areas where the non-survivors had been hit.
I prefer another example, though. It’s to do with cats, especially those falling from buildings. Clearly, I’m not a cat person.
You may have heard the idea that cats falling from six story buildings end up with fewer injuries compared to cats falling from lower buildings. Researchers who seemingly have plenty of spare time actually published this in scientific journals. I’d always heard this was because the height of taller buildings allows the falling cat more time to align itself, feet-first, reach terminal velocity, and prepare to land in a safer position.
That published study noticed that as the cats fell from increasingly higher buildings, their injuries tended to increase, until the buildings got up to six or seven stories. At that height, the reported injuries seemed to plateau off and even reduce.
Amazing. Proof that higher buildings are better for falling cats.
There is, though, another possibility. The study recorded injuries of those cats that had been brought into the veterinary hospital after falling from a building. The issue is that people don’t tend to bring dead cats into the vet hospital, such as those cats that fell from buildings seven, eight or thirty stories high. This was the reason why those researchers were seeing a higher number of injured kitties who had fallen from shorter buildings, and the reason why that original study conclusion was an example of survivorship bias.
So, how does this relate to that confident guru you love? He’s a multi-millionaire entrepreneur success story, and he hustled his way to greatness. We could assume then, that his hustling led him to success, right?
Not necessarily. He may have just been lucky and landed a few big clients or been in the right place at the right time. He may have done 13 things right and 26 things wrong, but those 13 right things paid off more at the time than the 26 things cost him (those same mistakes might cost someone else differently, or even be enough to sink their business). It’s also possible that he did everything right at the right time, but those actions may not be repeatable because of changed environmental factors.
The bottom line is that his anecdotal success can be inspirational and motivational, but it should not be considered strong evidence. His retrospective anecdotes shouldn’t be confused with prospective evidence. Prospective evidence is the sort of information that helps you confidently predict the future outcome of actions you take today.
In the CORE Marketing Method, I touch on the importance of The Hierarchy of Evidence. I use a pyramid model to explain the concept.
This model asserts that your Experience-based Assumptions are numerous and come immediately to mind, but they aren’t highly reliable chiefly because of survivorship bias and other logical errors. Consensus among a lot of people – the Wisdom of Crowds – is a little more reliable as it can wash out some bias (careful though, as groups produce some of their own biases).
Better again is the Wisdom of Experts, which is the consensus of people with demonstrated expertise (such as published academics, or respected business commentators who can point to real data, not just opinion). I recommend entrepreneurs do their best to access this layer of evidence as it is easily accessed and generally reliable. You may wonder how to discern who is an expert, and who is just a self-appointed guru? A good rule of thumb is that those genuine experts refer to actual data rather than war-stories, they have the respect of other similar experts, they never try to put on a hard-sell, their successes have been tested and repeated in different settings, and they often have a proven track-record in either the corporate world or formal educational institutions. Their wisdom is easily accessed through books, lectures, TedTalks, podcasts and blogs. Follow these women and men on LinkedIn and you’ll soon get a sense of what themes can be banked on and inform your business choices. In the CORE Marketing Method, entrepreneurs spend 30 min at the start of their week “filling up the reservoir” in general self-education, and these sources are the perfect place to find these bite-size portions of wisdom building.
Even more reliable again is to run a specific Bespoke Test in your own business based on good methodology, while an aggregated series of these tests on a single question is even better (Meta-Analysis). At the top, are Randomized Controlled Trials. These formal, academic research projects are increasingly becoming part of corporate big business, and when they’re published they can provide highly reliable data for the small business entrepreneur if the studied concept relates closely to your business question. Their real power lay in their ability to provide prospective information that robustly predicts future outcomes.
In reality, no small business owner will afford the time and money for their own RCT, but you can always read published RCT conclusions and find correlations for your business. That said, these highly reliable findings are a bit of a tough slog to wade through and are often filled with academic jargon. A more accessible way to the really useful advice is to find the clever people who author these RCTs, and then read their popular books, or check out their interviews on podcasts and videos. You’ll soon learn what their studies have proven.
All of this is the least sexy thing that aspiring entrepreneurs want to hear, but it’s simply too important to ignore. Most businesses will fail due to making poor choices (bad bets) based on poor information (their own biased assumptions, or the advice of gurus).
In full transparency, relying on hard data doesn’t match my own personality, either. My eyes light up at bold aspirational ideas and I prefer to plough my way through obstacles with optimistic gumption and crossed-fingers. But I’ve learned from the most unexpected place that relying on good evidence makes sense and is one of the keys to reducing risk.
That unusual place is the healthcare industry, where I spent about 15 years marketing medical brands to doctors and healthcare systems around the world. In this work, I came to realize that while doctors are often wonderful people and are almost always smarter than the average person, they seemed especially exposed to bias. This has been well documented in the healthcare profession and is one of the reasons that medical students don’t get to sling a stethoscope around their shoulder until they learn the maxim “evidence-based medicine”.
In a nutshell, the medical professional is trained to base their decisions not on gut-feel, not even on years of clinical experience and personal expertise, but primarily on whatever the evidence says. This means treatment choices should be made on the highest level of extensively tested, peer-reviewed, published science. Often, this is written into treatment guidelines that your doctor follows when they attend to your ‘flu (it’s probably just a cold). The rule of following evidence-based medicine has undoubtedly saved countless lives and helped against all manner of bias, survivorship being just one of them.
Witnessing this reliance on solid evidence in the world of healthcare, along with my nerdy interest in our biases, led me to understand the importance of relying on real data when making important decisions. Relying on insights mitigates against biases and will prevent choices being made on faulty assumptions.
Because it has been shared a bunch of times and liked by ten thousand followers, that Instagram video of your entrepreneurship influencer probably falls into “Wisdom of Crowds” on the Hierarchy of Evidence (which is to say, fairly low down on the scale). If enough people like it, try it out, and it seems to ring true, it may be worthwhile. However, if something higher in the pyramid contradicts it, it’s probably not worthwhile.
In reality, your favorite guru may be a nice guy, and he probably offers some general concepts that are harmless enough. In fact, it may be that the motivation that he evokes in you is useful in its own right. Just don’t mistake these general concepts and motivations as Rules, Laws, or Evidence of anything much at all.
For those decisions that shape the direction of your business and your life as an entrepreneur, you want to bank on something real. Actual evidence.
*snackable video is possibly my least favorite phrase. I only include it in this article to prove to millennials that I know what digital media is. Sort of.