It’s always a dilemma about what to do with the failed test cases during execution. Unfortunately, some test cases always fail. There are several reasons for that: the related feature is not mature enough, some problems happen after implementing a new or modifying an old feature, etc. If you have lots of test cases, then more of them will fail.
What to do?
You can execute the test knowing that it fails
You can skip the failed test cases
Neither is good enough. Using the first solution, it is difficult to detect which new bugs are present. If you know that formerly 8 test cases failed, and now 10, then you should find the 2 new ones among these 10. Sometimes it can occur, that a formerly failed test passed and a new one failed. In this case, you may not observe the new bug.
Skipping the failed tests is a better solution, but not perfect. Let’s assume that you have a long e2e test case that fails in the last step resulting in a minor bug. After code modification, this test would fail in another step resulting in a major bug. But as you skipped the test you will not recognize this.
What is a better solution? The answer is simple: execute each failed test case to the test step when it still passes, but the subsequent step would fail. In this way, all your test cases pass, therefore, if, after modifications any test fails, you can recognize it easily. This is also true for the shortened tests. They may fail to show a new bug within the same test.
If the related code part is fixed then you can delete or move the ‘end test case’ marker.
Harmony has the necessary feature. You can insert an ‘end test case’ where the step precedes the failed step. In the figure below one of the test steps failed:
We select the previous one ‘add items >= 30’ to set the end test case:
After clicking on ‘End test here’ the steps following it will be logically deleted:
The modified test case has to pass:
When the bug has been fixed, you can delete the end test case status in the same way:
留言