Stealing from Science

The Helix Nebula is 700 light-years away from Earth. The original image was taken by ESO’s VISTA Telescope.

I just finished the short but very powerful book A Briefer History of Time, by Stephen Hawking and Leonard Mlodinow. Space and how the universe works is something that has always fascinated me. I think it’s all the unanswered questions and the urge to understand what it ALL means, and the wonder of what may be out there.

A Briefer History of Time is a result of feedback about the original book A Brief History of Time in which many people requested a more accessible version. I for one am happy this was done as it’s now a great introduction to Hawking’s (and other’s) work which was relatively easy to grasp. Now that I have a taste for it, and a slightly better grasp, I’ll continue on and read his other works.

As I read non-fiction books I often think about lessons that can be misappropriated and applied to software testing. This wasn’t my specific goal while reading this book, but one that lingers in the background all the time. It’s often been discussed that testing can take many queues from science, and specifically the scientific method. I’m not going to try and put a new spin on this, but did have some ‘light bulb’ moments when reading this book that I wanted to share.

Here are some of the excerpts and how they impacted me:

“Any physical theory is always provisional, in the sense that it is only a hypothesis: you can never prove it. No matter how many times the results of experiments agree with some theory, you can never be sure that the next time a result will not contradict the theory. On the other hand, you can disprove a theory by finding even a single observation that disagrees with the predictions of the theory. As philosopher of science Karl Popper has emphasized, a good theory is characterized by the fact that it makes a number of predictions that could in principle be disproved or falsified by observation. Each time new experiments are observed to agree with the predictions, the theory survives and our confidence in it is increased; but if ever a new observation is found to disagree, we have to abandon or modify the theory. At least that is what is supposed to happen, but you can always question the competence of the person who carried out the observation.”

If you’re content with being confident, then great… but I think confidence can blind us. In testing there will always be an element of confirmation. We want to know that a product works, and continues to work. However, only seeking to confirm can be dangerous. We should also try and ‘disprove our current theories’, i.e. try and identify how the product does not work. While you look to confirm the product mostly works, also focus on changing the test approach and see what other important information you can identify. Don’t let confidence blind you into thinking ‘it’s all good baby!’

As Hawking mentions, when disproving the theory you can always question the one who does. So when you’re identifying ways the product may not work be sure to understand the testing you’re doing and why you’re doing it. You’ll often have to advocate and/or defend your work when raising bugs or questioning the value of the products and its features. Record and gather evidence so your bug advocacy is easier, and stakeholders understand their ‘latest theory has been disproved’.

Remember, you cannot prove that the product works 100% of the time… but you can prove it doesn’t. Check out one of Michael Bolton’s posts on the subject.

“Because the partial theories that we already have are sufficient to make accurate predictions in all but the most extreme situations, the search for the ultimate theory of the universe seems difficult to justify on practical grounds. (It is worth noting, though, that similar arguments could have been used against both relativity and quantum mechanics, and these theories have given us both nuclear energy and the microelectronics revolution.) The discovery of a complete unified theory, therefore, may not aid the survival of our species. It may not even affect our lifestyle. But ever since the dawn of civilization, people have not been content to see events as unconnected and inexplicable. We have craved an understanding of the underlying order in the world. Today we still yearn to know why we are here and where we came from. Humanity’s deepest desire for knowledge is justification enough for our continuing quest. And our goal is nothing less than a complete description of the universe we live in.”

When testing I’ve often been asked to ensure 100% code coverage, 100% requirements coverage, or to perform ‘complete’ testing of the product.

In my opinion what Hawking is witting about above is cost versus potential value. Will all the extra effort actually add value? In the quest for answers about the universe that is a particularly hard question to answer, and it can also be very difficult to answer when testing a product! For me, focusing on risk helps with this. When considering the testing that has been completed so far, how comfortable are we? Think about what risks may still be present and what impact they may have if we don’t go any further, or don’t focus on a particular part of the product while testing.

It’s important to remember that whatever risk remains is owned by our stakeholders. They may try, but in the end should not pass those decisions, nor that responsibility on to us (the testers). We inform but should not approve or sign off on releases, or the dreaded ‘quality gates’ that are often put in place.

It can be hard to get stakeholders to understand that we could potentially test forever and still not find all the bugs, but we can talk about finding many of the important bugs. In my experience stakeholders better understand the subject of risk than they do testing, so bring it back to that.

Again, check out some of Michael Bolton’s posts on when you may want to consider stopping your testing (here with a follow-up here).

“To picture this, suppose someone on a train bounces a Ping-Pong ball straight up and down, hitting the table twice on the same spot one second apart. To that person, the locations of the first and second bounces will have a spatial separation of zero. To someone standing beside the track, the two bounces would seem to take place about forty meters apart, because the train would have traveled that distance down the track between the bounces. According to Newton, the two observers have an equal right to consider themselves at rest, so both views are equally acceptable. One is not favored over another, as Aristotle had believed. The observed positions of events and the distances between them would be different for a person on the train and one beside the track, and there would be no reason to prefer one person’s observations to the other’s.”

“I found a bug.”

“That’s not a bug.”

Or…

“It was correct in the last build.”

“No, it’s correct now, it was wrong in the latest build.”

In either scenario (or many others you could dream up or may have experienced) both parties could be right. When testing we need to consider and be open to multiple perspectives, and perceptions for that matter.

However, unlike the scientific theory that Hawking writes about, in testing we often can’t just say both are right and move on. We’ll often have to decide who is ‘more right’ given the context of the product. Which ‘right’ will add more value to the people who matter the most? To answer that you’ll need to consider what problem the product is attempting to solve and who for, as well as the business goal of it. It can be a murky area with multiple levels of politics, so choose your ‘battles’ wisely and learn quickly when you ‘let it go’.

There were more paragraphs like these, but the above paragraphs had a very strong impact as I was reading them. As I read more of his work I hope to have similar reactions… and if I do I’ll share them

Peace…

by
David Greenlees