Testing Research Software

This poster describes the work presented in a published paper [1], which reports the results of a survey of research software developers. The focus of the study was to understand how research software developers design test cases, handle output challenges, use metrics, execute tests, and select tools. Specifically, the study explores the following research questions; RQ1: What are the characteristics of the research software testing process? RQ2: What challenges do developers face throughout the testing process of research software? RQ3: How do research software developers design test inputs? RQ4: What specific challenges do research software developers face when determining the expected output of test cases? RQ5: What metrics do research software developers use to measure software quality and test quality of tests? RQ6: How do research software developers execute their tests? RQ7: What testing tools do research software developers use and what are their limitations? RQ8: What features should a testing tool specifically developed for research software contain? RQ9: Are demographic characteristics of research software projects related to the testing process? The poster will contain graphs reporting the detailed results related to these questions. Our overall findings show that research software testing varies widely. The primary challenges faced by research developers include test case design, evaluating test quality, and evaluating the correctness of test outputs. Our findings highlight the need to allocate more resources to software testing and provide more education and training to research software developers on software testing. References Eisty, N.U., Kanewala, U. & Carver, J.C. Testing research software: an in-depth survey of practices, methods, and tools. Empir Software Eng 30, 81 (2025). https://doi.org/10.1007/s10664-025-10620-6