I test, therefore I am

A couple of days ago I had a thought going “While testing, I’m not just testing the software under test. What else am I testing?”

It quickly became clear to me that it’s not only things within the box but also thoughts and ideas that we test. So I started the following list, the reason behind it being that if I’m aware of something I can focus on it and test it. I may think of additional heuristics that may help me test this particular part and find risks, bugs, issues and information.

What do we test? 

• The application
• The operating system, anti-virus and other unrelated software running in the background
• (Test) environments and infrastructure
• Ourselves, our
  • Models, thoughts and ideas
  • Capabilities
  • Skills
  • Knowledge
  • Oracles
  • Biases
  • Feelings
  • Assumptions (thanks to Thanh Huynh)
• Needs – Testers vs Customers vs Business (thanks to Dan Billing)
• Our processes
• Our document structure, content and formatting
• Our peers, test partner or debrief partner
• Relationships with
  • Software and hardware
  • Project and non-project colleagues
  • Customers

If you can think of anything else, please add a comment and I’ll add it to the list!

 

 

Can we actually count bugs?

A recent quote on Twitter got me thinking:

I thought about the different ways we measure quality and one of the most infamous ones is to count bugs. I then though about if I actually really understand what it means to count. As a kid I used my fingers to help, and I thought I knew what I was doing. I’m a wee bit further than that now, meaning I’m not so sure anymore.

So I looked up the the definition what counting really is, from a mathematical point of view:

Wiki: Counting is the action of finding the number of elements of a finite set of objects.

OK, the element part is clear, we know what a bug is, right? Here’s one definition, again from the Wiki:

A software bug is an error, flaw, failure or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways. Most bugs arise from mistakes and errors made by people in either a program’s source code or its design, or in frameworks and operating systems used by such programs, and a few are caused by compilers producing incorrect code.

There are different definitions around, for example from IEEE 829, ISTQB, Softwaretestingfundamentals, RST, etc, the list is nearly endless. Let’s assume, and that assumption is a big one because it hasn’t happened so far, we all agree on the same definition of what a bug is. OK, we may be able to do that within one company at least if we don’t ask everyone and squint a bit while doing it.

So the question becomes, can we define a set properly? If we look at the definition above it’s actually pretty vague, for example who defines what incorrect or unexpected is? What’s unexpected to one person may not be to the next. With that definition we can’t actually define the element or a proper set as we’re dealing with relationships here and we’d need to define all unique elements of those relationships in order to define the set which is impossible.

To make matter worse, the set of objects has to be finite. So while we could, in theory, count all the grains of sand on a defined stretch of a beach we can’t count bugs as they rely on models, relationships, behaviours or ideas – which are infinite.

In short, since we can’t properly define the set which also has to be finite we can’t call it counting, at least not in a mathematical sense which is what most people do. We may point to the screen and count the number of bug reports in a bug tracker but that is a completely different kettle of fish.

Mistaking these two can and is being done for political reasons but if you’re serious about software testing we have to make sure people understand what it is exactly we’re reporting.

In other words, we can’t measure the number of bugs in a system but we can discuss them.