On the Implementation of OS-Specific Tests: The CPython Case

Modern software systems are frequently developed and tested across multiple platforms (e.g., Windows, Linux, and macOS). In the software testing context, practitioners adapt the tests to run differently according to the target platform. These tests, which need to identify the platform on which they will be executed, are referred to as OS-specific tests. In this paper, we present an empirical study to evaluate how developers implement OS-specific tests in CPython, which is the reference implementation project for the Python programming language. Then, we mine this project and assess their OS-specific tests quantitatively. For this, we propose three research questions to assess the frequency, location, and issues related to OS-specific tests. Our results show that OS-specific tests are common in the CPython project, and 13% of the analyzed test files are OS-specific tests (RQ1). OS Identification APIs are used more frequently in test code (53.46%), and the test decorator @unittest.skipUnless is the most used to skip tests depending on the platform (RQ2).We also find 170 issues related to OS-specific tests in CPython, and Windows is the most targeted platform (RQ3). Lastly, we discussed practical implications for practitioners and researchers. Based on our findings, we emphasized the importance of testing across multiple platforms and examined the relationship between issues and OS-specific tests, among other insights.