“Real knowledge is to know the extent of one’s ignorance” - Confucius

I have yet to meet a Software Engineer who thinks they’re bad at their job. Yet bad code full of bad smells, and failed projects abound, which seems to strongly indicate a degree of delusion the majority of us must be guilty of. Either we live in blissful ignorance of our own failings, or the small minority of Software Engineers who are bad are far more productive than those that aren’t.

There is an interesting observation that can be made about the opinions of drivers that has many parallels to that of Software Engineers. A study of drivers by Sweidish psychologist Ola Svenson found that in the USA 88% of drivers think that they are safer than average and in Sweden it is 77%. The only comforting thing about the former overestimating their ability, is at least they’re unlikely to squish you late at night after having a few. (Unless they made a mistake in the code of the self driving car they were programming.)

In social psychology there is a term called illusory superiority which is a condition of cognitive bias wherein a person overestimates their own qualities and abilities in relation to the same qualities and abilities of other people. This is one of the nicer biases to have (what person does not want to feel good about themself?) but also something which one has to be mindful of; in being over confident in our skills we open ourselves up falling into a failure of our own devising.

An example of this is the Dunning–Kruger effect which is where people overestimate their ability to complete a task. They found that a person’s inability to judge their own incompetence derives from their lack of knowledge of the correct standards for the given task. This means that you may make mistakes in a task, but you will not recognise them due to an inability to recognise this as a mistake, thus leading to an inflated idea of ability. They demonstrated this hypothesis by giving a cohort of undergraduate students a test on logical reasoning. Those in the bottom 12th percentile overestimated their performance and placed themselves in the 62nd percentile. Conversely, those in the top percentile underestimated their abilities. It is easy to believe in yourself as a great engineer, if you have no frame of reference!

One particular example of overestimating ability and knowledge amongst many others sticks in my mind. A particularly stubborn colleague of mine who had taken on the responsibility of building a reporting system. He’d designed a solution which was highly extensible and, as a piece of engineering, it was reasonably good. However, by bullishly not listening to any other opinions or advice about the issues that were becoming increasingly obvious: the solution was incredibly complex and took days to learn to configure, as well as it having several mistakes that meant the end report it produced was not accurate. By ignoring the clamour of voices around him pointing out obvious flaws in the plan, he had gone from one small mistake to a big mistake. The project was dropped and ended up being replaced.

Having perspective and being mindful of one’s lack of knowledge is difficult; it means examining our own deficiencies. There is a favourite story of mine from the book series ‘The Hitchhiker’s Guide To The Galaxy’ which particularly springs to mind when discussing this topic. One of the main characters, Zaphod Beeblebrox, was put in the Total Perspective Vortex. This was a machine built with the intention of showing beings the infinity of creation showing them just how unimportant and insignificant they are, thus leading to those subjected to it going insane.

“The machine was originally created by its inventor Trin Tragula as a way to get back at his wife. She was always telling him to get a “sense of proportion,” so he showed her the Vortex. Tragula was horrified to learn he had destroyed her mind, even as he proved his point that if life was going to live in such a vast Universe, one thing it could not afford to have was a sense of perspective.” - Douglas Adams

There is some element to this that is in many ways true: we live with inflated ideas of our own importance and knowledge; it helps us live a lesser version of ignorance is bliss. There is so much knowledge out there; more than we can possibly conceive of as (annoyingly) we have our cognitive limits. Just look up Dunbar’s Number for a good example of this. (For those who don’t like googling, it’s about how cognitive capacity limits social group size: the larger the primate brain size and the larger the average social group size. Humans have a number of 150. I imagine an AI brain could be infinite.)

We’re all guilty of overestimating our own knowledge, and this is why it is important to be mindful of this, and always seek perspective. By making pragmatic reasoned decisions based upon collective experience, we can build simple solutions and plan room for failure. There is nothing wrong with failure so long as you plan for its possibility, and learn from the experience. Ignorance may be bliss, but knowledge is power.