We compare the three classes of model-based testing techniques: stateless, stateful, and action-state. We show that stateless solutions may find fewer bugs with more test cases. We also compare the approaches of one and two-phase model-based testing. Our conclusion is that the best solution is the two-phase solution with action-state testing.
For practical systems mainly linear test design techniques are applicable. Roughly speaking, a test design technique is linear if the number of test cases is doubled when the implemented code is doubled. We have seen that covering requirement specifications is usually not enough. We consider the all-transition, (all-edge criterion) as a minimum. We investigated all three classes of modelling:
Stateless
Stateful
Action-state
The result is that by applying stateless testing, we only obtain a medium-level quality code. Using stateful testing, after bug-fixing, the code becomes high-quality. Using action-state modelling the code becomes even better. Obviously, we cannot draw any consequences from one example, however, investigating other examples by applying the test-design.org website, we got similar results.
Here is a table to summarize the results considering all three classes. For all cases, the all-edge criterion was applied.
Model class | stateless | stateful | action-state |
Ratio of the bugs found % | 80 | 93 | 100 |
Number of bugs haven’t been detected | 3 | 1 | 0 |
Number of test cases | 19 | 15 | 8 |
Number of test steps | 103 | 76 | 46 |
Test efficiency | medium | high | very high |
Defect prevention efficiency | medium | high | very high |
Coding | necessary | necessary | unnecessary |
Graphical modeling knowledge | necessary | necessary | unnecessary |
Table 1. Comparison of the model classes based on the car rental example
Given the requirements of GPT-4, it generated 13 test cases, the number of test steps is 46 and the defect detection efficiency is 67%, the worst of all solutions. Testers don’t need to be afraid to become jobless.
We can see that using the linear all-edge criterion is almost always appropriate for stateful and action-state solutions. Considering all these facts, we can conclude that action-state modelling is the way the best, stateful is in the middle, and stateless modelling is the weakest among model classes.
Now, let’s compare the work needed for ad-hoc or exploratory testing. We prepared a small case study in which experienced testers tested the original Car rental exercise that is only slightly different from our example. After reading and understanding the requirements, the testers created the test cases. The monitoring application counted the number of test steps. The Car rental exercise also contains 15 mutants, and when a tester finished the testing process, the application showed the percentage of the killed mutants. Here is the result:
Table 2. Statistics of the bug revealing capability of the Car rental application
The result is that exploratory testing reveals even fewer bugs than stateless modelling. On the other hand, it’s much faster for the first time, but executing the tests every day will not be faster after a while.
Now let’s consider the MBT approaches. Currently, there are only one-phase and two-phase MBTs. Here is a table comparing these two.
MBT approach | One-phase | Two-phase |
Level of modeling | Low-level | High-level |
Size of graphs | Larger | Smaller |
Should be readable by | Computer (application) | Human |
Implementation-independency | No | Yes |
Defect detection capability | Medium | High |
Can be used for model classes | All | All |
Table 3. Comparison of MBT approaches
Based on this table we can conclude that two-phase MBT is way better than one-phase MBT. As both MBT approaches can be used for all the model classes our final conclusion is that the best method is to apply two-phase MBT with action-state modelling.
Comentários