Seeing and Hearing an AI's "Thinking" Process

We are digital artists and humanists collaborating to design an interactive exhibit that illuminates and sonifies the processing of AI systems. We take inspiration from an exhibit in “AI: Mind the Gap” at MIT's Museum that educates viewers about how models work to classify and respond to their input.[1] At this exhibit, museum visitors are invited to draw a picture of a face on a touchscreen, and lights through a window display processing activities while the system classifies the images and chooses text to describe the sentiment they convey. The exhibit invites experimentation and further reading about how more complex AI responses work, and it helps visually orient viewers to the computational processes of emotion recognition. We are building our own immersive exhibit to surprise, engage, and prolong people's encounters with supposedly "intelligent" machines, especially regarding a specific dimension of interactive encounter–the signals of uncertainty and multiple possibilities in the processing of a language model. We regularly encounter generative chatbots on our mobile devices that easily simulate coherent personalities but surprise us when suddenly they lose track of context or fail to cohere. Such ephemeral flickers hint at instability or uncertainty belied by the programmed confident tone of most AI responses. We want visitors to pause a moment over this specific problem, to see and hear it in flickers and noise (perhaps an unsubtle clank or blast of static) in an extended moment of constructive play. Like the MIT exhibit, ours will magnify and slow down the response to try to visualize the “thinking process” as it investigates multiple possibilities, makes false starts on the way to a resolution. Perhaps we will be able to experiment with tipping points–what contradictory dimensions in a prompt resolve differently based on a slight change in the text input? In exposing dissonances in the processing, we may invite an experience of the "uncanny" as discussed by Masahiro Mori.[2] We enlarge the physical space for a memorable immersion experience, building with a computer, lighting, and sound equipment in a walk-in space designed for virtual augmented reality experiences in our campus library. But we minimize the processing and budget required for the simulation to maximize long-term sustainability of the project. We build with current lab computers and simple lighting/sound systems (Arduino kits, small projectors, or campus studio theater equipment) for translating computational processing signals into simple but dramatic lighting and sound effects. We have begun experimenting with open-source small language models (SLMs), that is, models designed to be run on a local laptop or computer system (Ollama, Qwen, and others).[3] We give the models agency to access external systems via Model Context Protocol (MCP) to trigger lights and sounds in response to processing events.[4] We will adapt our processing together with our students this spring to turn this into a dynamic, educational art exhibit as we tweak the coding scripts “under the hood” to change the output. We hope to discuss our project in progress at DARIAH's annual event.