top of page
Search

Comparison of model-based testing techniques

Updated: Apr 3



We compare the three classes of model-based testing techniques: stateless, stateful, and action-state.  We show that stateless solutions may find fewer bugs with more test cases. We also compare the approaches of one and two-phase model-based testing. Our conclusion is that the best solution is the two-phase solution with action-state testing.  

For practical systems mainly linear test design techniques are applicable. Roughly speaking, a test design technique is linear if the number of test cases is doubled when the implemented code is doubled. We have seen that covering requirement specifications is usually not enough. We consider the all-transition, (all-edge criterion) as a minimum. We investigated all three classes of modelling:

  • Stateless

  • Stateful

  • Action-state

The result is that by applying stateless testing, we only obtain a medium-level quality code. Using stateful testing, after bug-fixing, the code becomes high-quality. Using action-state modelling the code becomes even better. Obviously, we cannot draw any consequences from one example, however, investigating other examples by applying the test-design.org website, we got similar results.

Here is a table to summarize the results considering all three classes. For all cases, the all-edge criterion was applied.

Model class

stateless

stateful

action-state

Ratio of the bugs found %

80

93

100

Number of bugs haven’t been detected

3

1

0

Number of test cases

19

15

8

Number of test steps

103

76

46

Test efficiency

medium

high

very high

Defect prevention efficiency

medium

high

very high

Coding

necessary

necessary

unnecessary

Graphical modeling knowledge

necessary

necessary

unnecessary

Table 1. Comparison of the model classes based on the car rental example


Given the requirements of GPT-4, it generated 13 test cases, the number of test steps is 46 and the defect detection efficiency is 67%, the worst of all solutions. Testers don’t need to be afraid to become jobless.

We can see that using the linear all-edge criterion is almost always appropriate for stateful and action-state solutions.  Considering all these facts, we can conclude that action-state modelling is the way the best, stateful is in the middle, and stateless modelling is the weakest among model classes. 

Now, let’s compare the work needed for ad-hoc or exploratory testing. We prepared a small case study in which experienced testers tested the original Car rental exercise that is only slightly different from our example. After reading and understanding the requirements, the testers created the test cases. The monitoring application counted the number of test steps. The Car rental exercise also contains 15 mutants, and when a tester finished the testing process, the application showed the percentage of the killed mutants. Here is the result:

Tester

Potential bugs found (%)

Number of test steps

67

144

73

100

80

81

93

142

53

47

87

52

67

79

67

63

53

36

60

57

Average

70

80.1

Table 2. Statistics of the bug revealing capability of the Car rental application

The result is that exploratory testing reveals even fewer bugs than stateless modelling. On the other hand, it’s much faster for the first time, but executing the tests every day will not be faster after a while.

Now let’s consider the MBT approaches. Currently, there are only one-phase and two-phase MBTs. Here is a table comparing these two.

MBT approach

One-phase

Two-phase

Level of modeling

Low-level

High-level

Size of graphs

Larger

Smaller

Should be readable by

Computer (application)

Human

Implementation-independency

No

Yes

Defect detection capability

Medium

High

Can be used for model classes

All

All

Table 3. Comparison of MBT approaches


Based on this table we can conclude that two-phase MBT is way better than one-phase MBT. As both MBT approaches can be used for all the model classes our final conclusion is that the best method is to apply two-phase MBT with action-state modelling.


44 views0 comments

Comments


bottom of page