I test, therefore I am

A couple of days ago I had a thought going “While testing, I’m not just testing the software under test. What else am I testing?”

It quickly became clear to me that it’s not only things within the box but also thoughts and ideas that we test. So I started the following list, the reason behind it being that if I’m aware of something I can focus on it and test it. I may think of additional heuristics that may help me test this particular part and find risks, bugs, issues and information.

What do we test? 

• The application
• The operating system, anti-virus and other unrelated software running in the background
• (Test) environments and infrastructure
• Ourselves, our
  • Models, thoughts and ideas
  • Capabilities
  • Skills
  • Knowledge
  • Oracles
  • Biases
  • Feelings
  • Assumptions (thanks to Thanh Huynh)
• Needs – Testers vs Customers vs Business (thanks to Dan Billing)
• Our processes
• Our document structure, content and formatting
• Our peers, test partner or debrief partner
• Relationships with
  • Software and hardware
  • Project and non-project colleagues
  • Customers

If you can think of anything else, please add a comment and I’ll add it to the list!

 

 

Advertisement

Can we actually count bugs?

A recent quote on Twitter got me thinking:

I thought about the different ways we measure quality and one of the most infamous ones is to count bugs. I then though about if I actually really understand what it means to count. As a kid I used my fingers to help, and I thought I knew what I was doing. I’m a wee bit further than that now, meaning I’m not so sure anymore.

So I looked up the the definition what counting really is, from a mathematical point of view:

Wiki: Counting is the action of finding the number of elements of a finite set of objects.

OK, the element part is clear, we know what a bug is, right? Here’s one definition, again from the Wiki:

A software bug is an error, flaw, failure or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways. Most bugs arise from mistakes and errors made by people in either a program’s source code or its design, or in frameworks and operating systems used by such programs, and a few are caused by compilers producing incorrect code.

There are different definitions around, for example from IEEE 829, ISTQB, Softwaretestingfundamentals, RST, etc, the list is nearly endless. Let’s assume, and that assumption is a big one because it hasn’t happened so far, we all agree on the same definition of what a bug is. OK, we may be able to do that within one company at least if we don’t ask everyone and squint a bit while doing it.

So the question becomes, can we define a set properly? If we look at the definition above it’s actually pretty vague, for example who defines what incorrect or unexpected is? What’s unexpected to one person may not be to the next. With that definition we can’t actually define the element or a proper set as we’re dealing with relationships here and we’d need to define all unique elements of those relationships in order to define the set which is impossible.

To make matter worse, the set of objects has to be finite. So while we could, in theory, count all the grains of sand on a defined stretch of a beach we can’t count bugs as they rely on models, relationships, behaviours or ideas – which are infinite.

In short, since we can’t properly define the set which also has to be finite we can’t call it counting, at least not in a mathematical sense which is what most people do. We may point to the screen and count the number of bug reports in a bug tracker but that is a completely different kettle of fish.

Mistaking these two can and is being done for political reasons but if you’re serious about software testing we have to make sure people understand what it is exactly we’re reporting.

In other words, we can’t measure the number of bugs in a system but we can discuss them.

W5H3-A model for communication in software development

I first gave a talk about my model for communication at DEWT6 (Dutch Exploratory Workshop on Testing) in January 2016.

At the core is a very extensive mindmap which is intended to model all the different aspects of communication before it takes place in the context of software development, especially software testing. My examples are geared towards this area but it works for completely different areas as well.

The idea is for an upcoming conversation, meeting or other communication event to go through this model in advance and take notes (if needed) in order to improve communication.

People at DEWT suggested that they can see it being useful for retrospectives as well – how you use it is up to you. Adding to it may feel right, removing some parts to make it easier to work with – work with the mindmap that is right for you. After the talk Zeger van Heese (@testsidestory), suggested adding “feedback” to the mindmap and how to ensure that what was said wasn’t understood in a different way. That makes a lot of sense to me so that’s in now, thanks for the contribution!

Click on the map or here to go the Biggerplate site if you’d like to download the map. Without it understanding the model will not be easy if not impossible.

It may be useful to start at the top right (Who) and then walk your way through it in clockwise direction. That way it’s easy to figure out Who one wants to communicate with, What should be communicated and so on.

Adverse factors is an area that lists a couple of problems why communication may fail and what to look out for, something to check your approach against once it’s about finished.

Here’s an example how you may want to use it:

“I want to talk to a Software Tester about the testing they’ve just done.”

• Who is clear in this example, the tester.
• What I want to talk about needs to be clearer in my head. It could be test coverage, approach they took, potential problems discovered, …
• Why do I want to talk to them about it? Do I feel unsure that they have the skills to cover all aspects? Maybe it’s an area that I don’t know well and want to learn something about. Maybe there are several aspects as to why I want to have this discussion. To me this is one of the most important but often overlooked areas. The 5 Whys technique may be used here as well to get to root reason for communication.
• When do we have a discussion, right now, in an hours time? Maybe schedule in a weekly or daily meeting?
• Where do we want to have this conversation? In my office? Do I sit with the Tester in their open space office? Do we go to a meeting room or even go offsite? How does the where impact communication in this case?
• How much do we communicate? If the Tester is new in the team there may be a need to communicate a lot more tacit knowledge compared to long time employees in the company. Do I want to get a high level overview or go into specific details?
• How many people are present for this conversation? Is it a 1:1? Would it make sense to ask other testers to join us and have a group discussion?
• How is the biggest part and needs a lot of attention. The how is where all the information from the previous questions is distilled into an approach. The participant(s) may be insecure but experienced, boisterous but newbies, feel more at ease in a structured environment or not, some people can be criticised easily but feel not taken seriously if that doesn’t happen and so on. Different tools can be applied and combined in order to reach the goal of this conversation. Feedback from the communication partner determines how the approach may need to be changed during communication.

Check the adverse factors node for anything that may have a negative impact that hasn’t been covered by the rest of the model.

The “NOT” heuristic

I used the NOT heuristic to both get more ideas and to validate the thoughts that I have.

I’d ask not only Who do I want to communicate with, but also who NOT? Why do I want to communicate with someon or why NOT? This way it’s less likely that certain aspects are missed and it’s easy and fast to do.

 

Transformational Leadership part II

In the first part of the Transformational Leadership blog I described the theory. This second part is an experience report with a couple of examples.

I’ll start with the pre-requisites linking back to the first part; Willingness to experiment, Trust, Patience, Passion

When I started a new job I had enough passion – of course I was eager to do my best in the new role. In the beginning I didn’t have to worry about Senior Managers being impatient, the new guy bonus helped there. That left trust and the willingness to experiment. The latter was OK with senior management but for the team to experiment I had to gain their trust first.

The situation was such that the existing team hadn’t had a Test Manager for over a year and they were somewhat disillusioned but hopeful that the new guy would stand in for their interests. They were experienced and knew what they were doing with a lot of domain knowledge that was completely new to me.

So I sat down with them as a group and individually and asked them to tell me what went well and what didn’t (the latter list was longer…). Problems ranged from test environments going down several times a day, no commonly used bug tracking system and builds being thrown over the wall on the group side. On the individual side some haven’t had payrises in years, wanted to learn about other parts of the system but were stuck and felt that their contribution wasn’t valued.

So first I just listened to their problems. I also sat down individually with each tester to watch them test. That way I could learn the domain and also see how they approach testing their particular area of the system.

Within a short time we sent out some  guidelines how to use the bug tracking system (explaining the why) and introduced a template (the how) so that all necessary information was included for the developers. I had a chat with the development manager about the quality of the builds and what kind of problems to find in the test environments were acceptable and which were not – basically test entry criteria. He in turn sat down with his team and told them that the throwing over the wall times were over and that I would reject builds if they weren’t up to scratch. He set up monitors naming and shaming the developers who broke builds for everyone in the big office to see which improved build quality remarkably.

That all didn’t really take very long. What was probably even more important than getting more stable builds is that I gained the trust from the test team. I trusted them to continue testing the builds with their experience and domain knowledge while I continued to make them look better and remove problems in their way.

Another approach was that I mapped the test environments servers and applications, compared to our production environment and went to my boss to ask for money to get more servers plus time to get it all installed and configured. This also helped to show that I was indeed listening to their problems and doing something about it.

Since a trust relationshiop was built at that point I sat down with each tester individually and asked about what they thought would be needed next to bring the team forward and how they could help. Everyone had their own drivers and motivation so I introduced coaching for some and mentoring sessions for others covering the Individual part of the Tranformational Manager aspects. In team meetings I involved everyone in decision making, for example which new bug tracking tool we should go for and which ALM (We went with Fogbugz and TestRail).It helped the team spirit that people felt that they could bring in their own ideas and also get them implemented.

By that time we got more people in (one problem was simply not enough people on the ground) and everyone helped out when the going got tough. When one tester spent a lot of time on the private mobile (which was unusual) I asked if there was trouble. He replied that his kids and also his wife was sick at home and he was worried. I sent him home as his mind was clearly elsewhere. The work still needed done but others in the team jumped in and covered his most important work so he could go home. That was great to see and a sign that the environment and team identity aspects were progressing nicely. With an environment working and team identity like that I rarely needed to ask for overtime. More often than not people told me that they’d stay longer or come in on the weekend to get something done in time because they knew how important it was.  Knowing that if they had an emergency they’d get help made sure there was a drive to give something back.

What also helped is that we held a couple of meeting not in the official meeting rooms but in a cafe/bar down the street which was much more relaxing and actually more productive as well. Creating a stress-free environment got the whole team more productive as no time was wasted for people to cover their backs.

On a different note we had a three strikes rule – make a mistake once, no worries, we learn from it but please share it. Make it twice and we have a chat how to prevent it in the future and what we missed when we first thought we learned the lesson. Making the same mistake three times was a case of wtf, where’s your head, are you overwhelmed, do you have personal problems, what’s wrong? Note that I never had to go to part three.

SIDE NOTE: I wish I knew of Jurgen Appelo’s Delegation Levels from his Management 3.0 Workout book, that would have helped enormously.

Pairing with developers was another big step which worked really well on several levels. Testers got an early view of the implemented new features. Developers had to actually get their code to a stage where they could demonstrate it (which worked well after a first few embarassing fails). And they learned what was important for each other increasing mutual respect. The value of this cannot be overstated, a lot of communication problems, delays, misunderstandings and rework disappeared due to pairing.

These examples conclude the transformational leadership post. If you have examples of yourself or want to comment where you see things differently please share.

Transformational Leadership

I followed two recent Software Testing Conferences on Twitter, Eurostar 2015 (I took part in the mobile deep dive day) and Agile Testing Days.  What struck me was the difference in management styles that were promoted in contrast to what most people experience in their day job.

This blog is about what I perceive to be one of the differences –                Transformational Leaders vs Transactional Leaders.

Starting with the latter the Transactional Leader is the typical ‘stick and carrot’ type. The goal is compliance promoted through rewards and punishment. The emphasis is on maintaining the status quo by rooting out apparent faults and deviations from the process driven way of working. Apparently this is supposed to work well in a crisis, same as micromanagement, I do have my doubts.

The goal for a Transformational Leader is creating a self-sufficient team that can handle any of the numerous changing aspects of modern software development projects. The underlying idea is that there’s more intelligence and ideas in a whole team than in a single person. The leader serves the team by providing guidance and removing obstacles.

The four components for the Transformational Leader according to Wiki are

• Idealized Influence (II)
• Inspirational Motivation (IM)
• Individualized Consideration (IC)
• Intellectual Stimulation (IS)

That view is very focused on the leader rather than what the changes mean for the team and and how to get there. So I’m using a slightly different model using the four aspects below. I used the original (sub-)points from Wiki but also added a few of my own.

• Environment – how the work environment has to change to reach our goal
• Individual – what changes for the individual (with help from the leader)
• Team Identity – promoting moral standards for the whole team that everyone can identify with
• Leader tasks – how the leader can help reach the team’s goals

These aspects form the framework that a Transformational Leader is aspiring to. Before we get there we have a few pre-requisites that have to be in place, otherwise the change (and leader) is doomed to fail.

These are not hard and fast rules, for example trust from the team and senior management will grow over time but a base level has to be present.

While implementing the changes and working with the team I learned a couple of golden rules over time.

Most rules are self-evident, a few had to be learned the hard way though. Hopefully someone will find them useful and will not step into all the traps I encountered along the way.

That concludes the first part of the Transformational Leadership blog. The next will describe a few examples based on my experience.

If you have any comments or suggestions how to improve any of the above please leave a comment.

An Introduction to Transactional Analysis

After a recent Twitter discussion about treating adults like children I started thinking about Transactional Analysis (TA) again and how important it was when I learned about it (and it still is). It helped me a lot in the day to day tasks as a Test Manager and in private life as well.

Luckily understanding the basics is pretty easy and with a bit of patience the learnings can be applied quickly, be it in a work environment at home or anywhere else where people communicate.

In TA social transactions are analyzed to determine the ego state of a person as a basis for understanding behavior.

TA uses the PAC ego state model

• Parent (learned behaviour from own parents or parental figures)
• Adult (objective appraisal of reality – strengthening the adult is the goal of TA)
• Child (using behavior from our childhood)

In this example the ego states are easily guessed, the woman in child state and the man in parent ego mode.

We constantly switch subconsciously between ego states but it is helpful to observe the current state not only in oneself but also in the partner of the social transaction. If I know the ego state I’m in I can take a conscious decision to change it. If I also know the ego state of my opposite I can influence it and steer the course of the conversation. Note that there are also substates, for example the parent state could be caring and nurturing or commanding and demanding. I’ll leave out these substates as this would become too complex for this blog.

There are no right or wrong states, it depends on the context. I can tell my son off for not doing his homework putting me in the classic parent mode and him in the child state. Or he can tell me off for being late potentially putting me in the child and him in parent mode. I say potentially because I don’t have to go into child mode, I can choose. Splashing about in a swimming pool we could both be in child mode which would be entirely appropriate in that situation.

Why is this important in the context of software development? When a project manager asks the tester how much longer it takes until testing is finally done in what ego state is he/she?

The PMs question could be a request for information (Adult -> Adult) or it could be a dressing down in waiting (Parent -> Child). Without context we don’t know for sure but the ego state we choose will influence the course of the discussion. Let’s play through this example.

“How much longer” instead of “how long” and “finally” to me indicates a disgruntled parent mode. Unfortunately a common reaction to someone asking in parent mode is to answer in child mode. If that happens the tester may become apologetic addressing the (perceived) hidden aggression in this way; in an adult ego state the tester may give a timeline and also address the perceived aggression and ask if there is dissatisfaction in an attempt to resolve the situation in a peaceful manner; in parent mode the tester may become equally aggressive and just push back.

If I’m aware of this mechanic I can take a step back and choose my ego state. The ego state of the discussion partner is not always clear but can be figured out quickly by asking leading questions. This is a lot easier with people we know better.

A little picture may help visualising the ego states.

In a work situation a Child <-> Parent setup is usually not a healthy discussion, ideally we’d try and move the ego states to Adult <-> Adult.

There can be crossed pathways as well, for example one starts with Adult -> Adult

“Can you please tell me what the time is?”

and the second person responds with Child -> Parent

“I know that I have to hurry, no need to put more pressure on me!”

If there are crossed paths no fruitful discussion can take place.

When we know someone better it’s likely that “Games” are being played. By that I mean ‘repeating sequences of transactions that lead to a result subconsciously agreed to by the parties involved in the game’.

Say an old married couple is going over an old argument where the result of the discussion is known already and the actors and texts (arguments) are defined. These games can be played in a variety of situations and the outcome will only change if one or both actors change their ego state.

This concludes this blog. Experiment with the PAC model next time you have a conversation and see if you can spot the ego states. I’d be happy to discuss further- please leave a comment or get in touch via Twitter, email, Skype.

A bug count in context

Counting bugs and creating metrics out of the numbers is a widely (mis-)used practice in software development. I understand where Michael Bolton is coming from and agree to a large degree. Apart from the obvious problems it also fosters bad behaviour as people try to game the numbers reducing their value. Of course there’s a however looming…

I’ll raise my hand and profess myself guilty of counting bugs in the past and creating metrics and diagrams out of it. For a variety of reasons explained below it looked like it was helpful.

I made the mistake of thinking of the bug count as information which it isn’t. It’s data. It becomes information with context and someone attaching meaning to it, right or wrong. I’ll try and make the point that the counts are meaningless but that the context information is actually the important part.

Here’s one such example where I tried to read meaning into the data. In a non-agile development project (with variations for company and type of project) I expect a relatively steep rise in the bug count, a couple of waves when new builds are coming in, bugs are fixed and new ones found with a tapering off towards the end. Note: In agile projects not all bugs may actually get reported but are fixed on the spot so it becomes even more meaningless.

So the picture may look something like this:

Looking at the graph I’m wondering what I’m missing (yes, context). Are these the total number of bugs? Or just the high priority ones? The numbers are going up and down in a certain way, is that expected? For example was there a new big feature in the build around 11/14/10? If not, did we get more testers on the project to explain the sudden rise? What about the drop?

In short the bug count doesn’t answer questions, it triggers questions which most likely could have been asked without it or are meaningless if we want to actually know about the quality of the product. A talk with a tester is more meaningful than looking at bug count data.

In a past project I had a very sudden bug count rise where the answer was that I assigned a new contractor to the project. She found plenty of problems within a very short time span showing her experience which is why I hired her. After a review of the bug reports and subsequent discussion with both testers it became clear that pairing with the new contractor was a good idea to ensure that the knowledge rubs off. I used the metric not to inform me about the quality of the product but as a trigger to find out something, that I could have had with a 5 minute chat with the new contractor.

That’s where I see one of the problems with looking at bug count data, replacement.  I could have spoken to the tester immediately instead of wasting it on looking at a graph first.

Another example – the drop at the end seems off to me. Again from experience something similar happened when we lost the connection to the database – a network error. Since I was the first one in we could get the problem fixed before the rest of the team arrived by looking at the bug count. Note that we later added Splunk for proper monitoring not only of the operational machines but for the test environment as well rather than relying on this crutch. This problems was caught as a lucky accident, not by design.

The important part here is not the bug count but asking the right questions and implementing the proper tools.

Counting bugs usually has a very low initial cost as it’s provided with most bug tracking tools. This is one of the reasons it’s so widespread in addition to how easy it is to read meaning into the numbers.

At the time I made it very clear to the test team that they are not judged based on the number of bugs they find or that this would even find a way into their reviews which is the case in some places. If everyone understands what data we have in front of us and more importantly, what it doesn’t tell us it can be a useful tool in the context of your project. If we remind us of our biases we are less likely to interpret something into the data that is not there.

Will I continue looking at the bug count? Probably but the emphasis is now on finding the real and context information.

To make sure everyone is paying attention, the graph that you see above is not a bug count graph but an edited user count graph. So just because it looks like what you’re expecting doesn’t mean that’s what it is. In short, the numbers are meaningless. Context information is what we really should be looking for.

Happy hunting, not for bugs but for the right questions to ask.

Making notes while learning

It’s been a while since I last blogged – mostly as other things in life took over and I needed the headspace to concentrate on non-testing stuff. In order to get back into the mood I re-read some older testing books but also a few new ones which are not exclusively about software testing plus a variety of blogs.

Here are a few examples from the ones I read in the last couple of months:

Tacit and Explicit Knowledge by Harry Collins

#Workout: Games, Tools & Exercises to Engage People, Improve Work and Delight Clients by Jurgen Appelo
The Cynefin framework in testing explained by Duncan Nisbet

In previous training courses I did one trainer mentioned that if we remember 30% of what he taught we all did well. I’m naturally cautious if someone gives me a percent number but it sort of feels about right. Because I didn’t want to forget the important parts after reading the books I started taking notes in OneNote (any text editor will do).

To  keep my notes arranged in a (for me) intuitive way I created a couple of documents; General Management, Testing and because it’s such a large area one for Management 3.0 for Jurgen Appello’s work. In each document I created sections and then pages where I keep a summary of book, article or blog that I think is useful to remember in a years time.

For the Tacit and Explicit Knowledge book, which isn’t an easy read as fellow testers have affirmed, I found it useful to learn the terminology of the book; I summarised what is tacit, explicit and ostensive knowledge, how does he define an action and a string and what are the enabling conditions of communication. Taking notes while reading the book helped understanding the rather scientific style of Harry Collins writing.

For other books and articles what often helps most, because it’s usually a summary in its own right is using diagrams or pictures. The Cynefin framework is really visual and with the picture and a few explanations underneath I’d be able to explain it as an elevator pitch. Sometimes a single picture like the one from Michael Bolton’s blog Oracles from the inside out  is enough to let me remember what it was about and show the model to someone else.

For the Management 3.0 workout book I created its own document and there are many different sections that couldn’t be compressed down into a single page. In addition to summarising the content I also created single pages with lists that would come in useful when using some of the examples in earnest. Adding a couple of my own ideas what would work or where I’d see problems or at least question marks means that I can take what I learned and apply it whenever I need to.

This way I can now delve into the different areas (documents) if I want to get inspired or only vaguely remember a specific model and look it up. The summary is usually enough for me to remember corresponding details as well so creating it was time well spent.

I hope you find this useful too.

The first post – an explanation

A few people may remember my site over at Googles Blogger. Unfortunately I ran afoul of Googles security processes as they were thinking someone is hacking the account when I was logging in using an IP Spoofer, Cyberghost, making my location Amsterdam instead of my home town in Germany. Long story short, even though I have my username and password Google support is denying access to the account and it’s lost to me. Needless to say I’m not happy, I’ll likely write a blog about details of this fubar in the near future and Google’s horrendous support practices. If you’re still on Blogger I’d urge you to rethink your choice, move somewhere else and just leave a redirect in place.

This site is a continuation of the old Blogger site. If you have any comments for any of the old posts, please leave them here.