Why self-confidence usually works — even when it’s obviously idiotic
Observing team dynamics in a room with high-powered people such as CxOs, senior bankers or lawyers etc. is interesting for many reasons. But perhaps the most fascinating observation one can glean is how effective self-confidence can be in influencing others.
Influencing others is particularly important as it is often directly linked to an ability to succeed. Being products of an eons-long evolutionary journey, our ability to influence others (or more broadly, “impact change”) is especially important as it significantly improves the chances of surviving & thriving in a chaotic & resource constrained world.
This article is an attempt to understand how confidence became the one trait that became so closely associated with this ability. And what that means for you.
The Competence Narrative
When people try to influence each other, usually a simple playbook is followed. Firstly, there is an emergence of opinions or viewpoints that differ in content or presentation (duh), these opinions then battle each other for supremacy, and finally the winning opinion seems to be followed by a number of other individuals.
There are no rules that the playbook above MUST be followed — for example we could decide that both opinions are equally valid and carry on with our days. But most groups instinctively prefer the idea of one dominant opinion/winner vs two competing theories of equal weight.
An evolutionary lens might help in furthering our understanding of why this is the case. I suspect our preference for a clearly dominant opinion/viewpoint arises from our innate desire to optimize for surviving & thriving; and optimization problems are built to lead to clear winners.
Within an evolutionary context, it makes sense to go with the viewpoint that is closest to the “truth” as this will help improve our chances of surviving & thriving. Here the truth is defined by that mix of visible (& invisible) patterns that secretly govern the universe. A view point that is closer to these patterns is more likely to predict the future with greater accuracy, and hence optimize for our survival.
If we define “competence” to be the collective term for being close to the “truth”, we could say that a more competent viewpoint will improve our chances of self-preservation.
The breakdown of narrative
Unfortunately, here is where things take an unexpected turn. While most of us think we are optimizing for competence, many observers end up optimizing for confidence instead — mistakenly assuming the two to be closely related proxies for one another.
Competence & confidence may have some co-relation, but it is weak at best. And there are a few structural reasons for this
1. Competence is domain-specific (i.e., a competent scientist, a competent farmer, a competent programmer) — a function of the fact that different underlying patterns can govern different domains; however, confidence is usually not (i.e., a confident person is usually confident when espousing views on science, farming or programming).
2. Even within specific domains, competence & confidence aren’t strongly correlated. This is because we are usually flying blind with respect to the underlying patterns that govern the said domain — no one has an manual for the universe’s patterns. Instead we tend make very personal judgement calls on competence based on observable markers (amount of money in bank, number of followers, number of citations etc.). These are good enough approximation in some instances, but hardly perfect.
3. Furthermore, confidence & competence do not track linearly as people expect them to (the 45-degree line in the graph). Instead, for any given competence level, there is usually a whole spectrum of confidence levels
4. Finally, it is likely that there are biological limits on how much competence can be achieved. Humans are born with a certain body that has physical limitation born out of our evolutionary dance. There’s no reason to assume that our human bodies (brain and all) are equipped to decipher ALL the hidden patterns that govern outcomes in the universe.
In other words, there’s a natural limit to human competence.
But there’s no such limit to human confidence. The chart above should actually be redrawn to show humanity’s ability to be infinitely confident.
Implications of the myth: Defending your self
Knowing that competence and confidence have a weak relationship can elicit two types of responses: “defensive” or “offensive”. In this section we deal with aspect that may help protect you.
A sort of cynicism about the world & especially hyper confident people is inevitable. Who can one trust if not the person loudly and confidently proclaiming that they can fix all your problems?
The right answer, of course, is to not fully trust anybody who is extremely sure of themselves — even experts. And while this may be anxiety-inducing, I do believe the world would be better off with a little bit more of this cynicism.
One caveat — be sure to not fall into the confidence trap yourself; I’m looking at you flat-earthers & conspiracy theorists. For example, I believe it is a good thing that conspiracy theorists don’t take the scientific evidence as the gospel truth. A level of cynicism against very confident people (including scientists) is good; but conspiracy theorists take it too far by becoming irrationally confident in their own half-baked versions of the truth.
And admittedly, the conspiracy theorist versions are invariably more internally inconsistent than the competing scientific narratives — a sure shot indicator that their theories are likely less “competent” than scientific ones.
I find the following chart quite helpful to think about the relationship between confidence & competence that I encounter in my own life.
Irrationally self-confident conspiracy theorists peddling things such as flat-earth are what I call dangerous idiots.
Certain insufferable scientists or asshole geniuses occupy the space for toxic leadership.
My point is this: the world would likely be better off if we didn’t trust dangerous idiots or toxic leaders as much as we seem to.
Implications of the myth: Changing the world
On the other side of this sentiment, knowledge of the irrational trust humans place on a strong confidence-competence ratio can help you “hack” the system for your own purposes. The world is full of more competent people who lose out to less competent folks on a regular basis. And the difference usually boils down to confidence — why not use this to your personal advantage?
To use the graph shown earlier — you can become one of the dangerous idiots (or toxic leaders). The world is usually sculpted by dangerous idiots or toxic leaders (as they appear so confident) — and while it may not be good for the world, it usually works out very well for the idiot/leader himself.
Of course, many will argue that over reliance on confidence does not always work! And that is true — indeed, there are domains of human activity where competence seems to edge out confidence, but they are quite limited in number and scope.
These confidence-proof domains also seem to follow a few rules:
1. Domains wherein outcomes related to different methodologies can be easily judged (a sort of experimental process to determine what the underlying pattern truly is) AND
2. there is inherent incentive for audience to deprioritize convenience/laziness and make an effort to find out the truth for themselves (i.e., knowledge of underlying patterns in a domain are important enough for the audience to overcome taking someone else’s word for things)
Examples include standardized test-taking, generating financial returns consistently, testing scientific hypothesis. I would not recommend over-relying on confidence in domains which satisfy these criteria.
HOWEVER, while these domains are a part of daily life, they’re usually a very small part for most of us. The domains we interact with often on a day-to-day basis don’t follow these rules.
This is especially true within domains where judging competence through objective criteria is near impossible — e.g., art, music, religion, etiquette etc. In fact, most cultural phenom we see was determined by people relying on self-confidence to push their viewpoint to the masses.
In my own life, knowledge of this dynamic has helped me take back control of my feelings & reactions from others to myself.
From a defensive side, knowing that there is a limit to competence AND that the link between confidence & competence is weak gives me conviction for my own views when faced with extremely self-confident criticism.
On the flipside, knowing that the winner in many situations is usually the one who is more confident pushes me to act in an exaggerated manner of self-confidence. You’d be shocked as to how often this “play-acting” is enough for me to get things done my way!
