An affect-recognition system is typically evaluated by how well its outputs match human labels on a standard dataset. Six categories, usually — happiness, anger, sadness, surprise, fear, disgust — sometimes a few more, drawn from a tradition that treats discrete emotions as natural kinds: stable, universal categories with recognisable signatures in the face, voice, or body.
The accuracy scores reported against this scheme say something precise. They say the system can reproduce the labellers’ judgments. They do not say the labels correspond to anything with a stable signature in the world.
Lisa Feldman Barrett’s theory of constructed emotion takes the stronger position. Discrete emotions are not transmitted from an inner state through a universal expression. They are assembled by the brain using core affect, learned concepts, and contextual prediction. On this view, “anger” is not a thing waiting to be detected in a face. It is a pattern a perceiver constructs, using their own conceptual repertoire, from a scene.
Barrett and colleagues’ 2019 review in Psychological Science in the Public Interest examines four decades of evidence on inferring emotion from facial movements and concludes that the assumed one-to-one mapping between emotion category and facial configuration does not hold up. Faces do not reliably signal what basic-emotion accounts say they should.
This does not make affect-recognition systems useless. It means that when such a system scores well on a labelled benchmark, the claim being validated is narrower than it looks. The system has learned to reproduce the labelling tradition. It has not, from its own resources, established that the emotions it is detecting exist in the form the labels assert.
The portable idea: methodology inherits the ontology of its training labels. Accuracy against those labels cannot settle whether the labels carve the world at its joints.
Sources
- Lisa Feldman Barrett. How Emotions Are Made: The Secret Life of the Brain. Houghton Mifflin Harcourt, 2017. ISBN 978-0-544-13331-0. Publisher page.
- Lisa Feldman Barrett, Ralph Adolphs, Stacy Marsella, Aleix M. Martinez, and Seth D. Pollak. “Emotional Expressions Reconsidered: Challenges to Inferring Emotion from Human Facial Movements.” Psychological Science in the Public Interest 20(1):1–68, 2019. DOI: 10.1177/1529100619832930.