I test, therefore I am

A couple of days ago I had a thought going “While testing, I’m not just testing the software under test. What else am I testing?”


It quickly became clear to me that it’s not only things within the box but also thoughts and ideas that we test. So I started the following list, the reason behind it being that if I’m aware of something I can focus on it and test it. I may think of additional heuristics that may help me test this particular part and find risks, bugs, issues and information.

What do we test? 

  • The application
  • The operating system, anti-virus and other unrelated software running in the background
  • (Test) environments and infrastructure
  • Ourselves, our
    • Models, thoughts and ideas
    • Capabilities
    • Skills
    • Knowledge
    • Oracles
    • Biases
    • Feelings
    • Assumptions (thanks to Thanh Huynh)
  • Needs – Testers vs Customers vs Business (thanks to Dan Billing)
  • Our processes
  • Our document structure, content and formatting
  • Our peers, test partner or debrief partner
  • Relationships with
    • Software and hardware
    • Project and non-project colleagues
    • Customers

If you can think of anything else, please add a comment and I’ll add it to the list!



Can we actually count bugs?

A recent quote on Twitter got me thinking:You can't measure quality but you can discuss it

I thought about the different ways we measure quality and one of the most infamous ones is to count bugs. I then though about if I actually really understand what it means to count. As a kid I used my fingers to help, and I thought I knew what I was doing. I’m a wee bit further than that now, meaning I’m not so sure anymore.

kid counting fingers

So I looked up the the definition what counting really is, from a mathematical point of view:

Wiki: Counting is the action of finding the number of elements of a finite set of objects.

OK, the element part is clear, we know what a bug is, right? Here’s one definition, again from the Wiki:

A software bug is an error, flaw, failure or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways. Most bugs arise from mistakes and errors made by people in either a program’s source code or its design, or in frameworks and operating systems used by such programs, and a few are caused by compilers producing incorrect code.

There are different definitions around, for example from IEEE 829, ISTQB, Softwaretestingfundamentals, RST, etc, the list is nearly endless. Let’s assume, and that assumption is a big one because it hasn’t happened so far, we all agree on the same definition of what a bug is. OK, we may be able to do that within one company at least if we don’t ask everyone and squint a bit while doing it.

So the question becomes, can we define a set properly? If we look at the definition above it’s actually pretty vague, for example who defines what incorrect or unexpected is? What’s unexpected to one person may not be to the next. With that definition we can’t actually define the element or a proper set as we’re dealing with relationships here and we’d need to define all unique elements of those relationships in order to define the set which is impossible.

To make matter worse, the set of objects has to be finite. So while we could, in theory, count all the grains of sand on a defined stretch of a beach we can’t count bugs as they rely on models, relationships, behaviours or ideas – which are infinite.

In short, since we can’t properly define the set which also has to be finite we can’t call it counting, at least not in a mathematical sense which is what most people do. We may point to the screen and count the number of bug reports in a bug tracker but that is a completely different kettle of fish.

Mistaking these two can and is being done for political reasons but if you’re serious about software testing we have to make sure people understand what it is exactly we’re reporting.

In other words, we can’t measure the number of bugs in a system but we can discuss them.

W5H3-A model for communication in software development

I first gave a talk about my model for communication at DEWT6 (Dutch Exploratory Workshop on Testing) in January 2016.

At the core is a very extensive mindmap which is intended to model all the different aspects of communication before it takes place in the context of software development, especially software testing. My examples are geared towards this area but it works for completely different areas as well.

The idea is for an upcoming conversation, meeting or other communication event to go through this model in advance and take notes (if needed) in order to improve communication.

People at DEWT suggested that they can see it being useful for retrospectives as well – how you use it is up to you. Adding to it may feel right, removing some parts to make it easier to work with – work with the mindmap that is right for you. After the talk Zeger van Heese (@testsidestory), suggested adding “feedback” to the mindmap and how to ensure that what was said wasn’t understood in a different way. That makes a lot of sense to me so that’s in now, thanks for the contribution!

W5H3 mindmap

Click on the map or here to go the Biggerplate site if you’d like to download the map. Without it understanding the model will not be easy if not impossible.

It may be useful to start at the top right (Who) and then walk your way through it in clockwise direction. That way it’s easy to figure out Who one wants to communicate with, What should be communicated and so on.

Adverse factors is an area that lists a couple of problems why communication may fail and what to look out for, something to check your approach against once it’s about finished.

Here’s an example how you may want to use it:

“I want to talk to a Software Tester about the testing they’ve just done.”

  • Who is clear in this example, the tester.
  • What I want to talk about needs to be clearer in my head. It could be test coverage, approach they took, potential problems discovered, …
  • Why do I want to talk to them about it? Do I feel unsure that they have the skills to cover all aspects? Maybe it’s an area that I don’t know well and want to learn something about. Maybe there are several aspects as to why I want to have this discussion. To me this is one of the most important but often overlooked areas. The 5 Whys technique may be used here as well to get to root reason for communication.
  • When do we have a discussion, right now, in an hours time? Maybe schedule in a weekly or daily meeting?
  • Where do we want to have this conversation? In my office? Do I sit with the Tester in their open space office? Do we go to a meeting room or even go offsite? How does the where impact communication in this case?
  • How much do we communicate? If the Tester is new in the team there may be a need to communicate a lot more tacit knowledge compared to long time employees in the company. Do I want to get a high level overview or go into specific details?
  • How many people are present for this conversation? Is it a 1:1? Would it make sense to ask other testers to join us and have a group discussion?
  • How is the biggest part and needs a lot of attention. The how is where all the information from the previous questions is distilled into an approach. The participant(s) may be insecure but experienced, boisterous but newbies, feel more at ease in a structured environment or not, some people can be criticised easily but feel not taken seriously if that doesn’t happen and so on. Different tools can be applied and combined in order to reach the goal of this conversation. Feedback from the communication partner determines how the approach may need to be changed during communication.

Check the adverse factors node for anything that may have a negative impact that hasn’t been covered by the rest of the model.

The “NOT” heuristic

I used the NOT heuristic to both get more ideas and to validate the thoughts that I have.

I’d ask not only Who do I want to communicate with, but also who NOT? Why do I want to communicate with someon or why NOT? This way it’s less likely that certain aspects are missed and it’s easy and fast to do.