Automated notation translation of ballet choreography using sequence models

Computer vision is constantly evolving, expanding its reach into diverse application areas. One such domain where computer vision technology remains underutilized is ballet, specifically in the notation of ballet choreography. Using dance notation systems such as Benesh Movement Notation (BMN) to record ballet choreography helps protect choreographers’ original works and preserves the dance heritage for current and future generations. However, there is a lack of automated approaches to translate ballet video data into dance notation. Therefore, we propose a notation representation and translation approach along with sequence models called BeneshED and BeneshFormer to achieve the translation of ballet video data into Benesh Movement Notation. The results indicate that our models achieve promising results with low error metrics. The top-performing model achieved a mean squared error of 0.01, a mean absolute error of 0.02, a root mean squared error of 0.12 and a mean absolute percentage error of 2.19 %. Additionally, the results revealed a high correlation with the ground truth data, with a coefficient of determination ( \(R^2\) ) of 0.87, Spearman correlation of 0.6, Pearson correlation of 0.93 and Matthew’s correlation coefficient of 0.93. This research work marks a first step in demonstrating the feasibility of automating the notation of ballet choreography using computer vision, enabling choreographers, choreologists, and dancers to leverage technology to preserve their dance heritage and legacy.