Leveraging the Wizard of Oz Method for Stimulating Design: Benefits and Challenges From an Application Case Study

Abstract This paper presents a comprehensive understanding of the use of the Wizard of Oz (WOz) method for studying emerging design tools. This method involves hidden human “wizards” simulating the functionalities of an undeveloped technology without the user’s knowledge, making it a valuable method for assessing technologies that are not yet mature or are too resource-demanding. Although WOz has been widely used in fields like human-robot interaction and autonomous vehicles, its application in design research is relatively novel, here integrating complex sketch-based inputs with CAD and 3D modeling workflows to produce real-time visual feedback. Through a case study aimed at stimulating design ideas generation, we applied the WOz method to simulate an AI design assistant based on designer’s sketches during 17 one-and-a-half-hour design sessions. We evaluated the success of this WOz simulation, and the results showed exemplary performance by the wizards in both time management and content creation, along with a strongly positive designer perception of the simulation, and a rich data collection to inform research analysis. The WOz method offers two major advantages: versatility across domains and tasks, and cost-effectiveness. However, it also presents challenges, including the workload on wizards and ethical concerns. We conclude with three key axes of recommendations for implementing WOz in design settings: training wizards for feasibility, clearly defining simulated functionalities and experimental protocols, and ensuring system credibility. These insights will support future design research anticipating technological advancements to study emerging tools.