Scenario Based Testing
Mark Kampe
1. Introduction
Testing is about assessing and developing confidence about the
quality of the product being tested. Good tests
shake out important bugs. Efficency in test development can be
measured by the number of new bugs found per hour invested in
new test development.
Imagine, for a moment, assume that:
- the unit test cases were based (at least in part)
on the component specifications (which were, in
turn, based on the requirements).
- the people who wrote the unit test plans made
a diligent and professional effort to ensure that
those plans would validate that the component met
its specifications and requirements.
- the people who reviewed those plans made a diligent
and professional effort to assess
the adequacy of those plans to ensure that
the component met its specifications and requirements.
- the test plan was competently implemented, and
all the problems it turned up were addressed
during unit testing.
Given those assumptions,
developing additional test cases based on the requirements
and specifications would not seem likely to offer a high
return on investment. Such test cases would
not so much be looking for bugs in the component as gaps
in the test plan ... and that is what the review was for.
There probably aren't very many (such) bugs left to be found,
and what few there may be are likely to be in obscure
corner cases.
If we want to improve and gain additional confidence about the quality
of our component we are going to have to find a new basis
for additional tests.
When we were discussing representations for requirements,
use cases were suggested as an alternative
to declarative specifications. Use cases (or user stories)
offerred a few compelling advantages:
- they are tied to real-world problems (concrete)
- they capture ways customers need to use the product (authoritative)
- they are easily validated (trust-worthy)
They may have even been the real-world sources from which
the declarative requirements and specifications were derived.
Such use cases can also become the basis for test cases.
2. Developing Testing Scenarios
Start with the original use cases:
- describe a situation in which the customer would need to
use the product.
- enumerate the steps in the process, the associated product
interactions, the actions the user would take, and their
expectations at each point.
- script that set of operations and verification of the
expected results at each juncture.
Carrying out such a scenario might assume that there was already
information in the system (e.g. about a particular customer,
order, or stock item). But defining new stock items, adding them
to the inventory, setting up new customers, and placing orders
are each use cases in their own right. We can construct larger
scenarios (sometimes called sagas which combine numerous
use case stories in long chains of inter-dependent test cases.
Such scenarios can become long and complex, and encompass almost
all of the functionality of the product.
Scenarios can be used in even more complex ways:
- multiple scenarios (representing interactions with
different users) can be run concurrently (in parallel)
to generate higher levels of load and flush out
problems that result from concurrent operations.
- the number, timing, and combinations of scenarios
can be generated at random to produce ever-changing
loads.
- additional scenarios can be created that include
user errors, component failures, resource exhaustions
and other disruptive events, to ensure that the system
handles them properly and continues to function.
- fixed scenario scripts can be replaced with formulae
for the generation of new scenarios (with
varying numbers and combinations of operations on
a varying set of objects).
3. Conclusion
Scenario based test cases are:
- relatively easy to develop
- complex, and therefore likely to find interesting problems
- realistic representations of actual use
- quite different from and complementary to traditional
one assertion at a time unit test cases