Allowing test execution to continue after failures: tool support and an industrial case study for fault-tolerant online model-based testing

Context<br/>Model-Based Testing (MBT) can be executed in either offline or online mode. While offline MBT pre-generates test cases that can be independently executed and resumed, online MBT dynamically interacts with the system under test by generating test steps on-the-fly based on runtime feedback. This adaptive exploration makes online MBT particularly attractive in industrial contexts; however, it also introduces a structural fragility: when an unexpected failure occurs during execution, the testing session often terminates prematurely, preventing further exploration of the model.<br/><br/>Objective<br/>The objective of this study is to improve the robustness and usability of online MBT by enabling execution to continue in the presence of localized assertion failures without masking genuine faults.<br/><br/>Method<br/>We propose two lightweight execution-level fault-tolerance heuristics: Retry Failed Node and Blacklist Failed Node. The heuristics were implemented in the GraphWalker MBT tool by extending its execution and exception-handling mechanisms and evaluated using a large-scale industrial MBT suite at Testinium.<br/><br/>Results<br/>The retry heuristic mitigates transient and flaky failures, while the blacklist heuristic enables continued traversal after persistent localized failures. Together, they significantly reduce premature termination of online MBT execution with minimal and bounded overhead.<br/><br/>Conclusion<br/>Fault-tolerant execution semantics are essential for making online MBT practical in real-world testing. The proposed heuristics improve execution continuity and support broader industrial adoption of online MBT.<br/>