A recent quote on Twitter got me thinking:
I thought about the different ways we measure quality and one of the most infamous ones is to count bugs. I then though about if I actually really understand what it means to count. As a kid I used my fingers to help, and I thought I knew what I was doing. I’m a wee bit further than that now, meaning I’m not so sure anymore.
So I looked up the the definition what counting really is, from a mathematical point of view:
Wiki: Counting is the action of finding the number of elements of a finite set of objects.
OK, the element part is clear, we know what a bug is, right? Here’s one definition, again from the Wiki:
A software bug is an error, flaw, failure or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways. Most bugs arise from mistakes and errors made by people in either a program’s source code or its design, or in frameworks and operating systems used by such programs, and a few are caused by compilers producing incorrect code.
There are different definitions around, for example from IEEE 829, ISTQB, Softwaretestingfundamentals, RST, etc, the list is nearly endless. Let’s assume, and that assumption is a big one because it hasn’t happened so far, we all agree on the same definition of what a bug is. OK, we may be able to do that within one company at least if we don’t ask everyone and squint a bit while doing it.
So the question becomes, can we define a set properly? If we look at the definition above it’s actually pretty vague, for example who defines what incorrect or unexpected is? What’s unexpected to one person may not be to the next. With that definition we can’t actually define the element or a proper set as we’re dealing with relationships here and we’d need to define all unique elements of those relationships in order to define the set which is impossible.
To make matter worse, the set of objects has to be finite. So while we could, in theory, count all the grains of sand on a defined stretch of a beach we can’t count bugs as they rely on models, relationships, behaviours or ideas – which are infinite.
In short, since we can’t properly define the set which also has to be finite we can’t call it counting, at least not in a mathematical sense which is what most people do. We may point to the screen and count the number of bug reports in a bug tracker but that is a completely different kettle of fish.
Mistaking these two can and is being done for political reasons but if you’re serious about software testing we have to make sure people understand what it is exactly we’re reporting.
In other words, we can’t measure the number of bugs in a system but we can discuss them.
ObServant Tester 
Pingback: Testing Bits – 3/27/16 – 4/2/16 | Testing Curator Blog
I’m not sure I follow the argument of not being able to count bugs.
What’s the value of playing with arbitrary definitions and then claiming that they are not enough?
I can count bugs. I know them when I see them, as does everyone else. In some cases, What I consider to be a bug isn’t considered as such by others,and then we can have a discussion. A bug is some behavior of the product that is deemed a bug by the people in charge of that specific product. In some companies it is useful to separate bugs from change requests, in others it is not, but every team has a firm understanding of what is considered a bug in their product(s).
I can count bugs the same way I can count the number of chairs in a room full with chairs, stools and sofas. There is no way to define a rule that will state “this is what a chair is” without including some stools or a sofa, or without excluding some proper chairs. The fact that something can’t be strictly defined does not mean you can count it.
I can also count really easily – one bug, two bugs, three bugs etc. Given enough time, I can also count to infinity (I currently am counting up to infinity – one increment every birthday, and I’m making a steady progress).
So, are we saying that counting bugs is meaningless? I can agree with that, but playing with formal definitions seems a bit off for me. People can, and do, measure the amount of bugs in the software, or, if you want, you can call it “estimating”. The result is similar.
LikeLike
Hi Amit,
thanks for commenting.
To adress your first question “What’s the value of playing with arbitrary definitions and then claiming that they are not enough?”
The definition of counting is not an arbitrary definition but a mathematical definition. I’d like to ask you to follow the link and read up on it, if you haven’t done so. My claim is that the way we are using counting for bugs does not satisfy that mathematical definition, which not everyone will be aware of, that’s why I wrote this blog post. I like to make sure that I fully understand what I’m saying and in this instance I found that I didn’t at first so I shared it with the community.
You also wrote “…every team has a firm understanding of what is considered a bug in their product(s).” If there would be a firm understanding there wouldn’t be a need for discussion. I may have a different outlook of what firm understanding is though, that’s where definitions are useful so that we can agree if or if we are not talking about the same thing.
“The fact that something can’t be strictly defined does not mean you can count it.”
That’s where I respectfully disagree. What you can count are bug reports as I stated further down in the article, no problem with that. You can call anything you wish counting but, and that is the important part, you can’t call it counting in a mathematical sense.
LikeLike
Hi, thank you for the reply.
Re-reading my comment, it came out more blunt than I have intended. I do apologize for that.
Anyway, the point I was aiming for was this –
Sure, the definition you gave is a very formal one, and I am not trying to dispute the correctness of it.
What I am wondering about is the value that we gain from applying such a formal definition to our situation. There’s a joke about a guy who gets carried off by the wind in a hot-air balloon, when suddenly he sees someone on the ground and asks him “can you tell me please where I am?” to which that person responds “you are inside a hot-air balloon, about 30 feet above ground level” (one version of this joke can be found here – http://www.poeticexpressions.co.uk/poems/Engineer%20v%20Manager.htm ). Applying the mathematical definition of counting to the situation of bug count seems to me a lot like this situation – it’s technically valid, but practically useless. All those philosophical definitions are inconsistent with the usual usage of “counting” (For instance, given enough time, I can count to infinity. it won’t do me good, but anyone can count any infinite countable set – that’s why they are called “countable”. https://en.wikipedia.org/wiki/Countable_set).
So is your claim about not being able to count bugs – the theory part is a nice idea to play with during your spare time, but for any practical need, bugs are countable the same way PI equals 3.1415926 (if you need more precision – http://www.geom.uiuc.edu/~huberty/math5337/groupe/digits.html ) neither of these is actually PI, but the difference it makes to any practical problem is negligible. As long as I can point and say “this is a bug”, it doesn’t matter to me that there is no strict definition of bugs, because I can define that “for the purpose of measuring, a bug is anything that the product manager agrees is a bug” or “a bug is any item in the bug tracking system that isn’t being rejected”.
So you *can* measure bugs. Whether you *should*, or what meaning can you get out of these numbers is another thing altogether. I don’t see a need to hide behind rigid definitions that say we can’t count it.
LikeLike
Hi Amit, thanks for the thoughtful reply.
The reason for me writing this particular post was one of curiosity. Often – not always – there are problems when using words that have a different meaning to the intended one. I found it curious that our way of using the word ‘counting’ is actually different to what it really means. So I wrote this post.
I agree with you that with the word ‘counting’ in most situations it doesn’t matter. What I find important is to that we’re aware of the difference and can take a conscious decision IF it matters in a given situation or not. In very many ways it’s a training exercise to look at words that are different from our intended meaning and find out where they come from – Epistemology in action.
LikeLike