An AI tool for scaffolding complex thinking: challenges and solutions in developing an LLM prompt protocol suite

Abstract This paper reports an exploratory study examining the interaction between a theoretical framework for Complex Thinking and AI (LLMs), in terms of its potentialities and constraints. The aim was to develop and conduct a preliminary pilot evaluation of a tool comprising a prompt protocol suite for use with an LLM, to scaffold Complex Thinking. The tool is designed for use by an individual or group in relation to a given Target System of Interest (i.e., a real-world system, a problem, or a concern), supporting the development of more complex understandings of such systems that can guide more effective and positive actions and decisions. We describe the process of developing a suite of prompt protocols for scaffolding particular properties of Complex Thinking and report on the outcomes of a pilot test evaluation with a set of users across different domains.