Inviting the customer into the tests (Sherlock's riddle)

 I encountered cases in which I am proudly presented with test plans that have great coverage: each requirement has a link to one or more test cases so nothing is lost.

OK, great, I say. And how does your customer use this feature?

What do you mean? They ask me.

You have this here feature, everything in the requirement is covered, and you added end cases, negative tests, etc. But how will the user use this feature?

Here I get a few types of answers. From: 'It doesn't matter. If I cover the requirements I cover the user's behavior' to a simple 'I don't know and it is not important enough to find out'.

This is exactly what Context-Driven Tests are about. You must know, otherwise, you are testing with your eyes covered.

Don't believe me? See a real-life example.

Therac-25:

Therac-25 was a powerful radiation therapy machine. It had 2 modes:
X mode (powerful X-ray) and E electron mode which was designed for therapy. The technician usually used the X mode, which was a high-energy mode. When they needed to use the E mode it happened that out of habit, they pressed the X mode button first and immediately corrected by pressing the E mode button.

The machine didn't finish the X mode setup and was not built for such a fast change. The result was that the patients got high energy X-Ray when low energy was intended.

There was an error message, but it didn't make sense, so the technician repeated the damaging treatment thinking it didn't do anything.

Five people died out of this bug.

If the company that built the machine (AECL) had tested it while simulating real scenarios, after studying how technicians use the system, it might have been avoided.

But let's leave the depressing thoughts and concentrate on something more amusing that also demonstrates the point.

A riddle:

Sherlock Holmes walked down the street. He saw a friend whom he had not met in a few years just leaving his house.

Sherlock asked him the ages of his daughters. The friend told him that the product of their ages (multiplied together) was 36. After thinking Sherlock was unable to find the answer and asked for another clue. The friend replied that the sum of their ages is equal to his house number. Still, Sherlock was unable to answer and asked for another clue. The friend told him that his youngest daughter had blue eyes, and suddenly Sherlock gave the correct answer. Explain how.

Think for a moment.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

How does the daughter's eye color help? It doesn't look like it does. Maybe it's the friend-suttled way to say that we have all the needed data?

I guess we could start by checking all the options of multiplying 3 numbers (that represent the daughters' ages) and result in 36:

Too many options (and thus the request for a clue).

But there is another piece of information - the sum is like the house number. Alas, we can't see it. Let's see the options anyway (based on the first calculation):

So where does it put us? Nowhere. We can't see the house number.

But wait. Let's think like the client, Sherlock in this case. Let's imagine we are in 19th-century London, standing in the street and talking to a friend.

Sherlock sees the house, he knows the number and still asks for a clue instead of calculating as a smart person would?

There is only one possibility: there is more than one answer. Thus the house number could only be 13, and Sherlock has 2 answers that fit this conclusion (daughters ages are either 1, 6, and 6 or 2, 2, and 9).

This is why Sherlock asks for a clue.

So what is the answer?

It's reasonable to assume that Sherlock took this clue, the youngest daughter's eye color, to mean that there was one daughter younger than the others. This is not the case with the ages 2, 2 & 9. The only possible solution is 1, 6 and 6.

At first, I thought the riddle can't be solved. But when one looks at the situation (Sherlock knows some of the data we don't) and takes into consideration that Sherlock is a very smart person, we understand how the riddle is solvable.

What if we don't have direct communication with the client, or there is a large user number?

There are more options, some will be relevant some not:

1. We can sometimes at least hear the customers that have issues, maybe talk to them, via these media:

• Customer support. Go to the CR section and talk to them. You will be surprised of what you will find.

• Search in forums that relate to your product. See posts about users' experience and even suggestions for new features.

2. As long as we don't risk privacy issues, we can monitor them by internal stats we can dd to the code and external analytic tools like Google Analytics.

3. The product owner usually has knowledge about it.

4. Focus groups could be considered.

5. Use your product! Ask all the people in the company, not only the ones in the technological roles, to install and use your product. Collect the impressions and potential bugs.

Conclusion

Not every bug will result in dangering people's health. But we do see, hopefully, the importance of inviting the user into the testing room.

We see that this riddle can be solved only if we see the client in front of our eyes, and we see that cases like the Therac-25 disaster could have a detected on time.