MOCHA: Model Optimization through Collaborative Human-AI Alignment

We present MOCHA, a novel interactive system designed to enhance data annotation in natural language processing. MOCHA integrates active learning with counterfactual data augmentation, allowing users to better align model behaviors with their intentions, preferences, and values through annotations. Utilizing principles from Variation Theory and Structural Alignment Theory, MOCHA (1) generates counterfactual examples that reveal key data variations and commonalities for users to annotate; and (2) presents them in a way that highlights shared analogical structures. This design reduces the cognitive load on users, making it easier for them to understand and reflect on the data. Consequently, this approach not only improves the clarity and efficiency of annotation but also fosters the creation of high-quality datasets and more effectively trained models.