In the realm of automated testing, outdated test cases can become a cumbersome obstacle. This article explores a proactive approach where, rather than delving into the debugging process to determine whether a test case has become obsolete or has failed due to a bug, we adapt and update it to ensure continued functionality. By doing so, we not only save valuable time but also enhance the test suite’s overall effectiveness.
Obsolete test cases are a real problem. Articles mention that you cannot automatically differentiate obsolete tests and test cases detecting a bug as both will fail. According to a case study around 80% of failures in regression testing are due to bugs in the software under test and the other 20% of failures are due to obsolete tests. Therefore, you should debug 25% more test cases to eliminate the obsolete test cases as well. Fortunately (as almost always) there is a much faster and easier way.
First, we should emphasize that tests never wear out i.e. the pesticide paradox is false. The truth is that test cases can only be obsolete, but it’s quite different from wearing out. If there is no change in the requirement specification, then the tests neither wear out nor become obsolete. If a requirement is modified, then some test cases that cover it can be obsolete.
Our advice to debug test cases to recognize as obsolete is similar to test debugging – never do this. The solution is requirement traceability. The original method is a matrix that maps the requirements and the test cases. Here is an example:
Req ID | Req description | Test case IDs |
R1 | Cars and bikes can be added | TC01 - TC08 |
R3b | Renting 3 cars, if the customer hasn't selected any bike previously, then one free bike is added. | TC02 - TC05 |
R5 | If the customer deletes the free bike, then no money discount is given. Adding the bike back the discount is given again. | TC03 |
When a requirement changes, the related test case can be considered as obsolete. We can assume that the modified specification requires new test cases, and therefore the old ones can be deleted. This means that if R3b is modified then test cases TC02 - TC05 should be deleted and new test cases for R3b are to be designed.
However, if a requirement can be covered with more test cases, then probably a single test case covers only a part of the requirement. In this case, you can fine-tune the requirement so that, if a part of the requirement changes, then only one or a maximum of two test cases become obsolete when the sub-requirement is changed.
However, a test case can cover more requirements. If one of them changes and the other is not then a good test case may be deleted. Fortunately, a better solution can be used in model-based testing. Instead of mapping requirements to test cases, let’s make traceability between requirements and a set of model steps.
Here is an example. The green lines are labels describing a requirement or a sub-requirement:
Here R1, R3b, and R5 consist of all the steps below them, thus R1 consists of all the steps in the figure. Let us assume that R5 changed so that ‘R5 deleting the free bike, an identical money discount is given’, then we should consider only the three steps below this label. Reading the steps it’s easy to recognize that we should reimplement only the responses, i.e. the outputs such as total price remained. The fixed model is the following:
In this way you can fix your obsolete sub-model and thus the related test cases to be valid with only modifying some model elements.
Okay, but what if R1 changes that is covered by 8 test cases, involving several steps? Fortunately, this requirement is the basis of the other requirements and thus it will not change so that all others remain. For example, if it changes so that only cars can be added, then the two other requirements become also invalid. On the other hand, if motorbikes can also be added, then the existing model and test cases remain valid only some new test cases (model elements) should be added.
The conclusion is that by applying model-based testing especially Harmony, the obsolete test cases can be easily fixed to work or should be deleted. No debugging is needed, you can transform your test cases at a model level.