How much data? Measuring the complexity of thematic coding tasks and the amount of training samples needed for supervised modeling

Traditional methods of thematic analysis are critical for deep, nuanced understanding of complex planning challenges, but the manual intensity of the process limits its applications to increasingly large datasets generated by planning processes. Hybrid, supervised methods of natural language processing offer a solution by allowing researchers to scale theory-driven topics by using traditional methods to first label a sample of the data. However, because the amount of labeled data needed varies across thematic analysis tasks, researchers and practitioners face uncertainty in the time, cost, and feasibility of applying scalable text analysis to planning research and practice. How much data needs to be labeled to achieve acceptable accuracy and how does this depend on the complexity of the task itself? This study proposes theory-informed models for measuring task complexity and defining the relationship between complexity and the number of labeled samples needed. Through the analysis of six thematic analysis tasks in 103 climate action plans (CAPs), we find that by only defining complexity from a codebook of phrases for a topic, akin to the process of manual thematic coding, it is possible to accurately predict the maximum attainable accuracy scores for any given training size and labeling a very small sample of data improves the accuracy of the prediction. This framework provides a way for urban researchers to determine the manual cost of extending a thematic coding task to larger scales before embarking on the process.