Taming Affect: On the Construction of Objectivity in Data Annotation Practices

Abstract Affective computing applications promise to decode human emotional expressions and provide objective insights into users’ affective experience, ranging from frustration and boredom to states of clinical relevance such as depression and anxiety. In this paper, I explore the contexts of development of facial expression recognition systems, machine learning systems designed to recognize changes in facial expressive behaviors and produce some form of meaningful knowledge about such expressions. Drawing on 20 interviews with engineers and annotators involved in developing or working with these models, this paper sheds empirical light on the socio-material practices through which practitioners articulate notions of objectivity, truth, and accuracy to make sense of affective data and validate their systems’ predictions. By exploring two common annotation schemes used to label facial expression recognition training datasets, the Facial Action Coding System (FACS) and the Valence, Arousal, Dominance (VAD) model, I foreground the situated judgments and negotiation practices required to make sense of affective data. Here, I juxtapose annotators’ account of data work – the non-linear interpretative practices that are necessary to ‘fit’ the data into FACS and VAD coding grids – to engineers’ perceptions and expectations around annotation. Here, I show how practitioners mobilize ideal notions of ‘subjectivity’ and ‘intuition’ to manage the uncertainty and ambiguity of affective data.