Unveiling Cognitive Biases in Software Testing: Insights from a Survey and Controlled Experiment

Biases are hard-wired behaviours that influence software testers. Understanding how these biases affect testers' everyday behaviour is crucial for developing practical software tools and strategies to help testers avoid the pitfalls of cognitive biases. This research aims to assess the extent to which software testers know the influence of cognitive biases on their work. Our study was conducted in two incremental steps: a survey and a controlled experiment. Firstly, we developed a questionnaire survey designed to reveal the extent of software testers' knowledge about cognitive biases and their awareness of these biases' influence on testing. We contacted software professionals in different environments and gathered valid data from 60 practitioners. The survey results suggest that software professionals are aware of biases, specifically preconceptions such as confirmation bias, fixation, and convenience. Additionally, biases like optimism, ownership, and blissful ignorance were commonly recognized. In line with other research, we observed that software professionals tend to identify more cognitive biases in others than in their judgments and actions, indicating a vulnerability to bias blind spot. To build on these findings, we performed a controlled experiment with 12 participants to investigate the behaviour and biases exhibited by humans when attempting to solve a hypothetical test problem. Through thematic analysis, we identified prevalent biases such as confirmation bias, pattern recognition and overreliance, sunk cost fallacy, and anchoring bias among participants. Additionally, we found that collaborative problem-solving was a prominent feature, often leading to biases like groupthink.