Understanding Complex Sketch Recognition Strategies for Intelligent Sketch-Based Design Tools

Abstract Despite recent advances in multi-modal AI tools (e.g., tools leveraging text-to-image models), there is a significant gap in the ability of such systems to be incorporated into complex design and engineering work. This gap is further exacerbated in contexts where sketch-based inputs are desirable due to the difficulty in recognizing freehand sketches or interpreting underlying human intent. To better surface requirements for emerging sketch-based AI systems for complex design context, we consider a case study involving architectural design; this is a domain for which, to our knowledge, there have been no architectural sketch-based AI tools that recognize freely produced plans or perspectives for downstream applications, including generating inspirational images. Using a Wizard of Oz experimental paradigm, we substitute the “tool” with human agents and conduct a lab-based study in which professional architects complete a design brief using this “tool”. Results demonstrate that human agents not only rely on visible sketch elements (i.e., lines) and architectural drawing codes, but also on their memory of previous lines and their knowledge of the design brief to comprehend perceived lines. In addition to gradually developing an understanding of the designed artifact, human agents also construct an understanding of the designer’s intentions. These activities are crucial for the agent to obtain a functional model of the designed object, beyond a purely topological and geometric perception model. Insights about this human workflow bring new potential techniques of sketch recognition for design tasks, informing the inclusion of new resources and software within AI tools.