Direct Manipulation of AI Text via Gestures : Exploring Gesture-Based Interfaces for Flexible Chatbot Interaction

This thesis investigates why current chatbot interfaces often feel rigid andinefficient for users, especially during tasks like drafting emails, andexplores an alternative interaction model. The research first gatheredinsights from semi-structured interviews and design probes to pinpointfrustrations with sequential, text-based chatbot use. These insightsinformed the development of a novel prototype that uses a Leap Motionsensor to capture hand gestures, allowing users to manipulate chatbotgenerated text directly. The prototype maps intuitive gestures (e.g., sliding ahand upward to formalize tone) to real-time text edits, effectively mergingconversational AI with direct manipulation. In user testing, participantsfound the gesture-driven interface highly intuitive and felt more in controlof the chatbot’s output compared to traditional typing. They could, forexample, instantly adjust a message’s tone or politeness with simplemotions, reducing the need for iterative text prompts. Participants alsohighlighted the need for dynamic text generation integrated with gesturecontrols. The results show that a gesture-based model can make chatbotinteractions more flexible and efficient, pointing toward a new multimodalparadigm for conversational interfaces.