IoT systems testing: Taxonomy, empirical findings, and recommendations

The Internet of Things (IoT) is reshaping our lives, increasing the need for thorough pre-deployment testing. However, traditional software testing may not address the testing requirements of IoT systems, leading to quality challenges. A specific testing taxonomy is crucial, yet no widely recognized taxonomy exists for IoT system testing. We introduced an IoT-specific testing taxonomy that categorizes aspects of IoT systems testing into seven distinct categories. We mined testing aspects from 83 primary studies in IoT systems testing and built an initial taxonomy. This taxonomy was refined and validated through two rounds of surveys involving 16 and then 204 IoT industry practitioners. We assessed its effectiveness by conducting an empirical evaluation on two separate IoT systems, each involving 12 testers. Our findings categorize seven testing aspects: (1) testing objectives, (2) testing tools and artifacts, (3) testers, (4) testing stage, (5) testing environment, (6) Object Under Test (OUT) and metrics, and (7) testing approaches. The evaluation showed that testers equipped with the taxonomy could more effectively identify diverse test cases and scenarios. Additionally, we recommend new research opportunities to enhance the testing of IoT systems. • Conducted a literature review of 83 primary studies to develop an initial taxonomy for testing IoT systems, comprising seven key aspects: testing objectives, tools and artifacts, testers, stages, environments, Object Under Test (OUT), and testing approaches. • Refined and validated the proposed taxonomy through surveys involving 16 and 204 IoT industry practitioners. • Conducted an empirical evaluation using two case studies and 12 practitioners for each to assess the taxonomy’s effectiveness. • Provided structured guidance for practitioners to navigate and apply the taxonomy effectively. • Discussed insights from the empirical evaluation and offered recommendations for practitioners and researchers. • Set up two public access points for professionals to continuously access and stay updated with our IoT systems testing taxonomy. The first is hosted on the Ptidej website, while the second is available in a GitHub repository, ensuring that the latest version, incorporating newly identified aspects, is always accessible.