Metamorphic Relation Patterns for Metamorphic Testing, Exploration and Robustness

ABSTRACT Metamorphic testing (MT) is an effective software quality assurance method; it uses metamorphic relations (MRs) to examine the inputs and outputs of multiple test cases. Metamorphic exploration (ME) and metamorphic robustness testing (MRT) are advanced additions to the MT literature: ME can enable better understanding, and thus testing, of systems; MRT can test software robustness without an oracle. Metamorphic relation patterns (MRPs) are abstractions or templates for multiple concrete MRs. The performance of MT/ME/MRT relies on the quality of MRs, but the identification of MRs is often a manual task requiring some knowledge of MRPs, creative thinking and a good understanding of the system. In other words, the effective identification of MRs remains a challenge and requires further study. In this paper, we propose formal definitions to describe the relationship between multiple MRPs in order to classify them. We also identify a series of new MRPs and propose a new MT framework that can guide the identification and application of MRPs. Based on our definitions, we present the notion of MRP family trees and further propose two trees by classifying previously published MRPs and our proposed MRPs, in order to make it easier for users to find their target MRPs for reuse, reference or inference. Finally, we report on case studies using the MRPs and our framework to guide the identification of MRs for MT/ME/MRT and successfully detect multiple violations.