Why Does State Transition Testing Fail?
At the SEETEST conference, I asked the audience who uses state transition testing. Not a single hand went up. That is hardly surprising, because the traditional approach has several problems. It is no coincidence that most blog posts on the subject recycle the same two or three examples, above all ATM authentication.
The core problem is that the technique was taken over from software engineering and computer science unchanged. In software engineering, we use state machines to model a system. In testing, we want something different: we want to find bugs. A model that describes a system well is not automatically a good basis for finding faults.
Why states are very important
State transition testing rests on the observation that most systems are not stateless. A sorting algorithm is stateless: the input fully determines the result. In most systems, however, the output depends not only on the input but also on the state the system is in. Take deleting a book from a shopping cart. Normally, only that book is removed. But if the system is in a "free children's book added" state, both the book and the free children's book may be deleted.
The best-known test design technique based on states is state transition testing. It requires knowing all the states and the transitions leaving each of them. In the state transition graph, the nodes are the states and the events (actions) are the edges. Once the graph is ready, various test selection criteria can be applied, based on different graph traversal strategies.
Problem 1: state explosion
It soon became clear that even the simplest FSM (Finite State Machine) produces too many states for real systems, so it only works in very special cases, such as the ATM example. The EFSM (Extended Finite State Machine) reduces the number of explicit states, but it needs guard conditions: an action can be executed from a given state only if certain conditions hold. Note that this does not make the system simpler. The complexity moves into variables and guards.
Problem 2: where do the states come from?
If you study books, articles, or blog posts on state transition testing, there is one thing you will rarely find: how to choose the states. Most examples are tiny, built on systems with a very limited number of states. Real systems have an enormous number of program states, which would make state transition testing unmanageable. The EFSM is a mathematical construct. It lets us reduce the number of states with guard conditions, but it offers no method for choosing them.
An experiment
Let's try an example and ask Claude Sonnet 5 to choose the states and explain why it chose them.
Requirements
Base data
Pizza fresca – €18
Pizza pollo – €15
Coke – €3
Beer – €4
R0 – Initial state. When the ordering interface is opened, the quantity of every item is 0.
R1 – Increasing quantity. The user can increase the quantity of any item by pressing the "+" button next to it.
R2 – Deleting an item (setting quantity to zero). Pressing the "X" button next to an item sets its quantity to 0. Partial deletion is not possible. If the user has 3 Cokes (one free, two paid) and presses "X", the quantity becomes 0, not 2.
R3 – First-time free beverage offer. When the total price of food items only (beverages excluded) reaches or exceeds €45 for the first time, the system offers the user a free Coke or a free beer by showing two related buttons.
The user may choose Free Coke or Free Beer at any time while the food total stays ≥ €45. If the user selects nothing, the offer remains.
The chosen free beverage is added to the beverage list, and the order displays "Free Coke" or "Free Beer" accordingly.
When "Free Coke" is selected, the existing paid Coke(s) and the new free unit are combined into a single row (1 free + N paid, N+1 units in total).
If the food total drops below €45, the offer disappears until the threshold is reached again.
R4 – The free beverage type becomes fixed. Once the user selects the free beverage, its type is permanently fixed. It cannot be changed from Coke to Beer or vice versa, even if the food total later drops below €45 and rises above it again. No new offer is made, and no beverage switching is allowed.
R5 – Withdrawal of the free beverage. If the user deletes food items and the food total falls below €45 while the free unit is present, the free beverage is automatically removed from the order. However, the system remembers which beverage was free.
R6 – Automatic restoration of the free beverage. If the food total later reaches or exceeds €45 again and the user has not deleted the free beverage, then no new offer appears, the user cannot choose a different free beverage, and the system automatically restores the same free beverage that was originally selected. This ensures deterministic behavior: the same beverage always returns.This refers only to the beverage's current free unit, not to a persistent flag. An automatic removal (R5) never counts as user deletion. Each time the free unit is restored, automatically (R6) or manually (R7), this status resets, and the next removal is judged independently, based only on how it happened.
R7 – User deletes the free beverage. The user may delete the free beverage manually with the "X" button. If the food total is still ≥ €45 and the user adds the same beverage type again, that beverage is added for free.If the food total later reaches ≥ €45 while at least one paid unit of that beverage is present, one existing paid unit converts to free (no new unit is added, the quantity stays the same). If no unit of that beverage is present when the total reaches ≥ €45, nothing happens automatically: the user must add a unit themselves, and R7's original clause decides whether it is free or paid at that moment.
R8 – Clear All. "Clear All" removes all items (food and beverages), removes the free beverage, and forgets the previously selected free beverage type. If the user then reaches €45 in food items again, a new free-beverage offer appears and the user may choose either free beverage again.
The untrained LLM's solution
This is the dialogue between me and the LLM.
Me: I'm attaching a specification. Later I will need tests generated according to state transition testing, but for now I only need the states and an explanation of why you chose them. Please answer in English.
Sonnet:
Modeling approach. I identified the state variables that change how the system reacts to the same event. The free-beverage rules (R3–R8) are the only stateful part of the spec. The food total and the quantities are data, not states.
The state variables are:
Food total vs. the €45 threshold (below / at or above)
Free beverage type memory (not chosen / Coke / Beer)
Whether the free unit is currently in the order
Why the free unit is absent, if it is (automatic withdrawal or user deletion)
Let B be the fixed free beverage type, Coke or Beer. States S3–S6 exist once per type, so the model has 6 abstract states, or 10 concrete ones.
ID | State | Food total | Type | Free unit in order | Offer buttons |
S1 | No offer, type not chosen (initial state, R0) | < €45 | none | no | hidden |
S2 | Offer open | ≥ €45 | none | no | visible |
S3(B) | Free beverage active | ≥ €45 | B (fixed) | yes | hidden |
S4(B) | Free beverage auto-withdrawn, restoration armed | < €45 | B (fixed) | no (removed by R5) | hidden |
S5(B) | Free beverage deleted by user, food ≥ €45 | ≥ €45 | B (fixed) | no (deleted by user) | hidden |
S6(B) | Free beverage deleted by user, food < €45 | < €45 | B (fixed) | no (deleted by user) | hidden |
Why these states
S1 and S2 (type not chosen). R3 and R8 make the offer depend on the threshold, so "below" and "at or above" must be separate states. The offer is visible only in S2. It disappears when the total drops below €45 and reappears when it is reached again, so S1 and S2 are connected in both directions. Clear All (R8) returns every other state to S1.
S3(B) (type fixed, free unit present). This is the state after a free beverage is chosen (R3) or restored (R6/R7). Its outgoing behavior is distinctive: no new offer, no type switching (R4), withdrawal when the total drops below €45 (R5), and deletion by the user via "X" (R7).
S4(B) (auto-withdrawn). R5 and R6 require the system to remember the type while the free unit is gone, and to restore the same beverage automatically when the total returns to ≥ €45. No other state behaves this way. Without S4, deterministic restoration could not be tested.
S5(B) and S6(B) (deleted by user). The spec distinguishes automatic removal from user deletion: "An automatic removal (R5) never counts as user deletion." Both removals leave the free unit absent, but they behave differently. After auto-removal (S4), the free unit is restored automatically when the total returns to ≥ €45. After user deletion (S5/S6), it is not restored automatically. Instead, adding B at ≥ €45 makes it free (S5), and reaching ≥ €45 with a paid B present converts one paid unit to free (S6 → S3).
S4 and S6 look identical from the outside (type fixed, no free unit, food < €45) but react differently to the same event, so they must be separate states. Splitting by threshold (S5 vs. S6) is needed because the transitions differ. In S5 the user can add B for free right away, while in S6 the conversion rule applies only when the threshold is reached.
Six states are not enough
Next, I asked Sonnet to generate tests:
Me: Please generate the tests based on these states. Apply the 0-switch test selection criterion.
The LLM generated 22 test cases, and they caught 21 of the 30 artificially injected bugs (70%). That is not a bad result, but nine bugs slipped through. My hypothesis is that two things are behind this: six states are far too few, and the basic 0-switch criterion is not strong enough on its own.
In the next post, I will show how to derive states more effectively.
Do you want to learn more? Book a 45-minute demo.
Comments