Sign2Vis: Automated Data Visualization from Sign Language

Data visualizations, such as bar charts and histograms, are essential for analyzing and exploring data, enabling the effective communication of insights.While existing methods have been proposed to translate natural language descriptions into visualization queries, they focus solely on spoken languages, overlooking sign languages, which comprise about 200 variants used by 70 million Deaf and Hard-of-Hearing (DHH) individuals.To fill this gap, this paper proposes SIGN2VIS, a sign language interface that enables the DHH community to engage more fully with data analysis.We first construct a paired dataset that includes sign language pose videos and their corresponding visualization queries.Using this dataset, we evaluate a variety of models, including both pipeline-based and end-to-end approaches.Extensive experiments, along with a user study involving 15 participants, demonstrate the effectiveness of SIGN2VIS.Finally, we share key insights from our evaluation and highlight the need for more accessible and user-centered tools to support the DHH community in interactive data analytics. 1 IntroductionData visualizations, such as bar charts, scatter plots, and histograms, effectively represent, analyze, and explore data while facilitating the discovery and communication of insights (Marriott et al., 2021).Despite the availability of numerous tools (e.g., Tableau's Ask Data (Setlur et al., 2019) and Amazon's QuickSight (qui, 2024)) and programming languages (e.g., Vega-Lite (Satyanarayan et al., 2016) and ggplot2 (Villanueva and Chen, 2019)),