Designing A Multi-modal IDE with Developers: An Exploratory Study on Next-generation Programming Tool Assistance

Researchers have envisioned and pioneered data-driven programming assistance for developers based on their interaction with the tools via multiple sensors such as eye trackers, microphones, and AI. However, these new sensors gather sensitive data from programmers, to what extent users can accept them and in what form they may work well are largely unclear. Meanwhile, developer tools such as static analyzers have been criticized for poor usability and not involving end users enough during their development. To make data-driven programming assistance design work, toolmakers need to partner with programmers.In this paper, we adopt a design process under the guidance of Participatory Design (PD) which aims to empower end users. Our design pipeline builds on a survey of 68 professional developers. The next stage is a design workshop with five participants sketching out ideas for alleviating the pain points reported from the survey. Based on these inputs, we then developed two types of tentative design representations consecutively – three conceptional designs and one low-fidelity prototype. Lastly, we tested the interactive digital prototype with five experienced programmers.From the user test, we learned that developers have more interest in gaze-driven assistance than voice- based assistance for reading and understanding code in an integrated development environment. Further, we report our hands-on experience with involving developers from the beginning to the end of this design process. This informs future work on using PD to study the development of developer tools.