Neither fair nor legal. How and why untrustworthy digital ecosystems evolve

Untrustworthy technologies create systemic harms for their users, often by design, and at the societal level. However, the nature of these technologies and the reasons why they evolve have not yet been well investigated. This research proposes a framework for conceptualising untrustworthy technologies as an ecosystem-level dark side phenomenon. Using a novel sensemaking model, we explore how values are encoded within a technology artefact through the contexts, conditions and ideologies of its conception, and how they may manifest as signals of untrustworthiness. We then illustrate the phenomenon through a critical realist, causal study of the dynamics that enabled and sustained the Robodebt social welfare scandal. We argue for the importance and necessity of moral evaluation of untrustworthy technologies, especially in the low optionality, high consequence settings of algorithmically determined social welfare programmes. This study enhances understanding of untrustworthy technology and its characteristics as a new entry to the dark side canon. The proposed model and techniques provide a foundation to guide future research in this critical area.