Wearable Haptic Music Player with Multi-Feature Extraction Using Spectral Flux and Yin Algorithms

Vibrotactile feedback synchronized with audio through haptic music players (HMPs) creates a synergistic effect that has been shown to improve the music listening experience. However, current HMPs are still unable to efficiently retrieve multiple music features, decelerating app scalability and jeopardizing long-term user engagement. This study introduces a wearable HMP that utilizes piezoelectric actuators and a novel audio-tactile rendering algorithm that uses YIN to extract pitch and spectral flux for rhythm. Building upon prior work, the system additionally features a modified discretization step and software optimization to improve multi-feature extraction and tactile display of music. The pitch, melody/timbre, and rhythm displays, respectively, were validated using Mean Average Error (MAE), Dynamic Time Warping (DTW) distance, and accuracy, yielding normalized averages of MAE = 0.1020 and DTW = 0.1518, and a rhythmic pattern accuracy of 97.56%. The Yin algorithm was shown to greatly improve the tactile display of vocals, with slight improvements for bass and accompaniments, while spectral flux and software optimizations significantly improved rhythm display. The wearable HMP effectively communicates multiple music features without the pitfalls of prior approaches. Future research can improve the system’s audio-tactile signal fidelity and explore the qualitative merits of multi-feature extraction in HMPs.