AI-Powered Test Design When AI Is Trained

Untrained AI can generate test cases from specifications, but this approach suffers from several critical flaws:

• ✅ They generate tests based on faulty specifications; when the input is flawed, so are the outputs.

• 🔍 They often produce incomplete test cases, typically missing initial setup steps.

• 🔗 Their test cases may rely on outcomes from other cases, violating the essential principle of test independence.

• ⚠️ Many software bugs are control-flow errors that require domain testing, yet untrained AI fails to recognize when such testing is needed.

  
  

Harmony’s AI-powered test design resolves all of these shortcomings. After an extensive training period, Harmony begins by analyzing the requirements holistically—checking for consistency, completeness, clarity, and correctness. Instead of blindly generating flawed tests, Harmony highlights problematic specifications and offers potential corrections.

Importantly, Harmony interprets requirements in context. When R1 influences R2, it integrates the necessary R1 steps into the R2 test case. But it doesn’t stop there—our tool elevates this practice by generating joint test cases that effectively validate both R1 and R2. In contrast, untrained AI treats each requirement in isolation, often neglecting prerequisite steps.

Classic test design approaches typically apply 2-value or 3-value boundary analysis to detect control-flow errors. Unfortunately, these are insufficient. For example, a rule like:

IF the price reaches 50 euros, THEN a 10% price reduction is given

requires four specific test cases to be properly validated. Harmony applies a refined domain testing strategy, as published in Modern Software Testing Techniques (co-authored by Attila Kovács). This method is not only highly reliable—it approaches optimal coverage.

Moreover, Harmony identifies which requirements truly demand domain testing—a task that even experienced testers struggle with.

Here is an example to demonstrate the differences between untrained AI and Harmony’s trained AI:

Form Editing Permissions — A Comparison

Consider the following set of requirements:R1 An existing form can be opened and editedR2 Anyone who creates a new form has admin rightsR3 The admin can grant admin, editor, or reader permissionsR4 The admin can revoke these permissionsR5 An admin can delete another admin or themselves, provided at least one admin remainsR6 Readers cannot modify the form, but admins and editors can

An untrained AI produced 16 verbose test cases. One example is:

Test Case R4.4: Admin Deletes Another Admin – FailureSteps:

1. Log in as Admin A

2. Select a form where Admin B is the only other admin

3. Attempt to remove Admin B’s admin rights

   Expected Result: The system prevents Admin A from removing Admin B’s admin rights. An error message is displayed.

   
   

This test case is deeply flawed:

• Step 1 assumes Admin A exists, despite no setup steps ensuring that.

• Step 2 presumes Admin B exists.

• Step 3 contradicts R5. If there are two admins, removing one should succeed, not fail.

In comparison, Harmony generated a much leaner and smarter set of just 5 test cases. Here’s one that overlaps with the flawed scenario above:

R1 Anyone who creates a new form has admin rights.

User creates a new form => User automatically receives admin rights for the form

R2 The admin can grant admin, editor, or reader permissions to anyone.

Admin grants admin permission to User A => User A receives admin rights

R4 The admin can delete another admin or themselves, as long as at least one admin remains.

Admin deletes User A's admin rights => User A is removed as admin => Original admin remains

Admin attempts to delete themselves as the only remaining admin => Deletion is prevented => Error message indicates at least one admin must remain

Lines beginning with R1, R2, R4 are labels for traceability.

This single Harmony-generated test case validates R1, R2, and R4 seamlessly. Two additional cases cover R2 independently as well.