Artifacts: A Quality Assurance Perspective of Fairness Evaluation for ML-enabled Systems

Abstract: Machine Learning (ML)-enabled systems are increasingly present in the lives of many people. The inherent use of these systems in various sectors essential to society has raised concerns within the scientific community regarding ethical issues and, more specifically, fairness. In this regard, many solutions exist to assess a system’s fairness or to help mitigate bias. However, there is still a gap in the practical evaluation of these technologies in real-world contexts. In this study, we selected a fairness evaluation checklist from the literature and applied it to six stages of the development cycle of ML-enabled applications in real-world scenarios. The goal is to identify and analyze the actions taken at different stages of development to mitigate bias and ensure fair applications. In addition to using and evaluating the selected technology in a real-world ML-enabled system development scenario, the main contributions and findings of this research include: (1) evaluating fairness at differentstages of system development; (2) through the results, identifying improvement opportunities for earlier stages in the development cycle and positive aspects to sustain and emphasize, thus contributing to better quality assurance and bias mitigation; (3) Additionally, observing how fairness assessment can be practically integrated into software development, thereby fostering interaction between Software Engineering (SE) and ML.