An Interactive Visual Enhancement for Prompted Programmatic Weak Supervision in Text Classification

Abstract Programmatic Weak Supervision (PWS) has emerged as a powerful technique for text classification. By aggregating weak labels provided by manually written label functions, it allows training models on large‐scale unlabeled data without the need for costly manual annotations. As an improvement, Prompted PWS incorporates pre‐trained large language models (LLMs) as part of the label function, replacing programs coded by experts with natural language prompts. This allows for the more accessible expression of complex and ambiguous concepts. However, the existing workflow does not fully utilize the advantages of Prompted PWS, and the annotators have difficulty in effectively converging their ideas to develop high‐quality LFs, and lack support during the iterations. To address this issue, this study improves the existing PWS workflow through interactive visualization. We first propose a collaborative LF development workflow between humans and LLMs, where the large language model assists humans in creating a structured development space for exploration and automatically generates prompted LFs based on human selections. Annotators can integrate their knowledge through informed selection and judgment. Then, we present an interactive visual system that supports efficient development, in‐depth exploration, and iteration of LFs. Our evaluation, comprising a quantitative evaluation on the benchmark, a case study, and a user study, demonstrates the effectiveness of our approach.