Against Interaction Design

“Against Interaction Design” distills a position that’s emerged over a decade of creating interactive art. As a manifesto, I first present it in its raw form. Next, in the discussion, I offer context, justification, and details on the alternative forms of human-computer interaction (HCI) it proposes.Look at the icons and words on my keyboard: ⌘ command, ˄ control, fn function, ⇧ shift. These describe the processes of a factory or a military unit, not a conversation, a dance, or friends eating together. What of the rest of being human?Behind every interface is a model world. People are modeled as profiles. Emotions are modeled as emojis 🤷. The interface defines how I can sense and shape that world. It defines my relationship to that world: what I can do, who I am, and who I can be.To interact through an interface, I need to think in terms of the model of that interface. I need to translate my vague, complex intentions into the language of that interface. I need to align my thoughts and actions to the structures and processes conceived by its designer. I need to see myself as a part of the model the interface implies.The more my interaction is designed, the more involved the designer becomes with what I can do and how and why I might do it. The more I conform to the designer’s ideal user, the less space I have to express the complexity of who I really am.An “expert” user is someone who transforms into the person the interface needs them to be. Those who struggle or refuse to adapt are excluded.This is not so much a problem of bad interaction design as it is a problem of what interaction design is.Marshall McLuhan argued that we shape our tools, and thereafter they shape us [1]. Except that it is not we who shape our interfaces. Interaction designers do.We all use the same few interfaces crafted by a handful of people. Each design decision, however heroic, however thoughtful, however detailed, affects us all in the same way. The more our interaction is designed, the more our agency is constrained to fit within the models conceived by the designer.The asymmetry of interaction design gives interaction designers too much leverage over our lives.When a single interface becomes standardized across so much of our lives, it makes us all share its singular model of that world. This works for standardized tasks like flying a plane or filing a tax return. It works for a game where we can freely step in and out of the role it defines for us.But it doesn’t work for human expression and social connection because it filters out, rather than amplifies, the chaos that makes us human.The singular model of a standardized interface centers what matters and sidelines what doesn’t. It replaces diversity with efficiency. It reduces our messy needs into explicit goals we can all agree on.But the rest of being human does not reduce. We must continually adopt and discard different world models to express our thoughts. A model is always an imperfect approximation, but what we try to express is real. There is no single model that can capture it all.A standard interface that spreads to define how we relate to each other, to society, to ideas, and our ability to express who we are? This is dangerous.A standard interface standardizes how we think. Some of humanity’s worst deeds emerge when the plethora of human wildness is filtered through a singular model of the world. Singular models become invisible. We forget how to discard them. Monoculture.When it comes to human expressivity, we cannot free people by designing how they can behave. But we can help people free themselves by letting them negotiate their own relationship with technology.We need interfaces that flex, adapt, and warp as needed by the human, interfaces that emerge through negotiation between human and system. We need interfaces as diverse as the people that connect through them.An alternative to designed interaction is negotiated interaction. Negotiated interaction aims to uncover interfaces that:Negotiated interaction supplants ⌘ command and ˄ control with ≈ resonate and ⚭ unite.“Against Interaction Design” is fueled by my experience in creating interfaces for dancers. Each time I invent a new form of gestural interaction, the dancer seems to lose some of their existing expressivity. Before my intervention, the dancer’s body was completely free to express through movement. Now, certain movements have consequences. The dancer must incorporate this external design into their kinesthetic potential. I believe that this diminishing effect of interaction design (IxD) is particularly visible in dance but generalizes to any interface.What of a document editor or the cockpit of a plane? Interaction design is clearly essential here. But these examples remind us of HCI’s original focus on the workplace, where computers were used for institutionally determined activities. From this, IxD has inherited a tendency to frame activity as the manipulation of abstract representations of real-life things (e.g., pages, posts, streams). In doing so, it fixes the ontological position of the designer [2]. Even a decision on the accessibility of a software feature defines a “landscape of permissibility” that enforces the designer’s opinions [3]. The asymmetry of this relationship mirrors the industrialized workplace, where people are marshaled from above to conform to a rational plan rather than forming their own plans. Anthropologist Arturo Escobar traces this “ ‘colonization’ of the lifeworld . . . [by] explicit calculation and theorization” to nineteenth-century modernism [4].Computers today have drifted from the workplace into social connection, political organization, and creative expression. These realms clearly rely on a diversity of expression. However, more subtle is the need for individuals and communities to define their own ontologies. For example, the relationship model of a social media platform may be an aggregation of atomic interactions—posts, friend requests, reaction buttons—that distort the unique relationship between two individuals. The asymmetry of ID means that ontologies are imposed from above rather than emerging from below.Many human pursuits are harmed by thinking in terms of abstraction and manipulation [5]. For example, equating social connection to manipulating social media posts reduces human relationships to data generation. It transforms the human into a “user,” another abstraction to be manipulated by the platform. Optimizing user behavior for the platform’s interest leads to toxic behavior [6]. Yet we seem trapped in discourses that blame toxic behavior on people rather than systems. Governments and platforms respond to toxic behavior not by freeing users from their dehumanizing structures but by ever more top-down restrictions and surveillance. We are so enmeshed in the industrial paradigm of HCI that it has become invisible.“Against Interaction Design” takes an extreme position because from there, one can see the absurdity of the extent to which IxD has “colonized the lifeworld.” By fundamentally rejecting IxD, it demands a radical alternative to how we think about human-machine relations. It proposes negotiated interaction but leaves open how it might be realized; its ethos is not to prescribe specifics. For illustration, I briefly share here one possible interpretation of negotiated interaction from my own practice.In Sonified Body, a collaboration with artist and researcher Panagiotis Tigas, we devised mappings from body poses to sound and invited dancers to improvise with the system [7]. To remove ourselves as gestural designers, we trained an unsupervised machine learning model, a variational auto-encoder (VAE), on motion capture recordings of one individual’s movement. This movement was not labeled nor preconceived but was 12 hours of free improvisation performed as a morning practice over one month.The VAE is a neural network that compresses and decompresses the body’s pose via its encoder and decoder, respectively. The model effectively factorizes the pose into a fixed encoder, which captures structures consistent across the dataset (e.g., the distances between the joints of the body), and an embedding, which captures the current pose given what the encoder already knows. In effect, the embedding is a representation of pose customized for one individual. It is optimized to maximally capture what the individual is doing right now rather than what the individual does in general.In Sonified Body, we use the embedding to control synthesis parameters. Neither I nor the dancer can control what mapping will emerge from the VAE. It defines a many-to-many, nonlinear relationship that defies explanation or reductive reasoning. Move any body part, and every parameter changes. And yet, the mapping offers smooth and consistent coverage over a 16-dimensional parameter space. One can learn it in an embodied way by finding a feel for it through play and exploration. From the task-based mindset typical of HCI, our process is absurd because the outputs cannot be anticipated. But we are working with improvised dance and somatic awareness. Our activity is open-ended: Material is encountered rather than envisioned. The system provides an open space for exploration optimized for our vocabulary of expression but, as far as possible, devoid of opinions about what we might do with it. To me, this combination of complexity and openness is much like human interaction. There is space for ambiguity, for sitting with the unknown and the vulnerability this brings.In my subsequent project, Self Absorbed, I arbitrarily connected the embedding to the latent parameter of the image generation model StyleGAN3 [8]. This creates a fully non-designed yet expressively powerful mapping from pose to image.The Sonified Body system is negotiated because the gestural language emerges from the individual’s movement language. But could we not instead empower the user to design their own gestural interaction? This would allow a kind of analytical negotiation, but this is not enough for embodied human interaction. Just as we cannot learn to ride a bike purely by analyzing the process, we cannot fully rationalize our embodied selves. In designing a system for myself, I am still constrained by my preconceptions. Unsupervised machine learning brings the potential for negotiation with the human as they are, not the human as they articulate themselves to be. McGilchrist frames embodied thinking and abstract reasoning as two mutually dependent yet conflicting modes of consciousness. The digital predilection for manipulating virtual abstractions has led to the latter over-dominating [9]. Rajko further argues that embodied thinking is more than a theory of cognition; it is a way of being in the world, a kind of conscious awareness rooted in sensitivity and empathy [10]. Dance-based interaction may seem niche, but consider how long it has been more common to maintain human relations by filling out website forms than by expressive physical movement.Against Interaction Design takes an extreme position because it is from there one can see the absurdity of how far interaction design has colonized the lifeworld. By fundamentally rejecting interaction design, my aim is to offer an alternative starting point to consider human-machine relations.