Identifying Basic Emotions and Action Units from Facial Photographs with ChatGPT

Abstract Recognising facial expressions of emotion is crucial for social interactions. In addition, identifying these expressions underlies the study of social cognition, consumer behaviour, and many other fields. Using automated facial coding (AFC), this task can be completed more efficiently while potentially benefitting human-computer interactions. Here, I explored ChatGPT’s ability to recognise expressions of basic emotions using high quality images for which human performance, as well as that of FaceReader (a commercially available software), had previously been collected. In Experiment 1, a forced-choice labelling task found that ChatGPT outperformed both humans and FaceReader in identifying the intended emotions from their expressions. Experiment 2 focussed on the facial action coding system (FACS), requiring ChatGPT to identify activated action units (AUs) and their intensities from these same images. The chatbot’s overall agreement with a FACS-certified specialist was at least as high as FaceReader, and was around the criterion required for human certification. Further, ChatGPT’s detection of a large subset of AUs showed good or very good agreement with the specialist’s coding, comparable with FaceReader’s performance. Finally, although overall intensity ratings showed relatively poor agreement with the human specialist, ratings for several of the AUs agreed well with human coding, especially if a small level of tolerance were acceptable. Taken together, ChatGPT’s ability to perceive human facial expressions and their AUs from high quality images provides a possible alternative to current AFC tools, as well as an interesting avenue for investigation regarding advances in human-computer interactions.