Experience: Bridging Data Measurement and Ethical Challenges with Extended Data Briefs

To promote the responsible development and use of data-driven technologies—such as machine learning and artificial intelligence—principles of trustworthiness, accountability, and fairness should be followed. The quality of the dataset on which these applications rely is crucial to achieve compliance with the required ethical principles. Quantitative approaches to measure data quality are abundant in the literature and among practitioners; however, they are not sufficient to cover all the principles and ethical challenges involved. In this article, we show that complementing data quality with measurable dimensions of data documentation and of data balance helps to cover a wider range of ethical challenges connected to the use of datasets in algorithms. A synthetic report of the metrics applied (the Extended Data Brief) and a set of Risk Labels for the Ethical Challenges provide a practical overview of the potential ethical harms due to data composition. We believe that the proposed data labeling scheme will enable practitioners to improve the overall quality of datasets and to build more responsible data-driven software systems.