Interactive visualisation of collaborative dynamics: a VLM-based approach for behavior and affect analysis

ABSTRACT: Collaboration is crucial in design and management, fostering innovation, problem-solving, and decision-making. We explore the use of vision-language models (VLMs) for analyzing collaboration, focusing on detecting social behavior and group affect. By fusing multimodal cues, VLMs enable more context-aware reasoning beyond surface-level perception. We develop a pipeline, a structured prompt and an interactive visualization for integrating VLMs into the analysis workflow. Comparing VLM and human analysis results, we discuss how VLMs can advance collaboration analysis and the remaining challenges.