SYNC: SYnergistic aNnotation Collaboration between Humans and LLMs for Enhanced Model Training

Large language models (LLMs) have demonstrated impressive performance across a wide range of natural language processing tasks, highlighting their potential as effective data annotators. While LLM-generated annotations tend to be costeffective, they are often error-prone and may inadvertently introduce bias. It is advantageous to harness the strengths of both LLMs and humans to ensure higher accuracy and reliability in the annotation process. In this paper, we present a multi-step, human-LLM collaborative approach to optimize data annotation for Stack Overflow datasets. We begin by applying TF-IDF to rank and prioritize relevant elements. Next, we utilize NLP Transformer and UniXcoder to leverage their deep contextual understanding for handling the code-related queries and discussions typical of Stack Overflow, resulting in more consistent automated labels. Finally, human annotators re-annotate to correct potential errors and mitigate bias introduced during earlier stages. To support human-LLM collaboration, we developed SYNC a research prototype that implements SYnergistic aNnotation Collaboration through an intuitive graphical user interface, enabling real-time interaction between human annotators and the LLM, allowing for iterative refinements. Overall, our approach is designed to integrate automated efficiency with human oversight to improve annotation outcomes, particularly for complex or domain-specific tasks such as those found in Stack Overflow datasets.