Explaining and Contesting Judicial Profiling Systems

This paper discusses how a right to an explanation can enable litigants to contest judicial profiling systems used in the administration of justice. Providing such explanations can, however, be hindered by the technical opacity of these systems, the presence of integrity concerns, or external private rights and interests. To overcome these obstacles, this paper argues that the exercise of the right to an explanation relies on technical and organizational obligations that require developers to make profiling systems contestable by design. Based on this conceptual framework, this paper critically interprets EU Data Protection Law, the right to a fair trial, and the new AI-Act. It shows how these laws can (partially) protect contestation, as well as their limitations and loopholes.