Instrumental Timbre Transfer Based on Disentangled Representation of Timbre and Pitch

Timbre transfer is a complex audio synthesis task that changes the distinctive perceptual attribute of one instrument to resemble another while preserving the underlying musical content. In order to investigate the possibility of effective representation for pitch and timbre, this paper explores recent advances in timbre transfer using deep learning models, and presents a framework for multiple-to-multiple instrumental timbre transfer using a beta-variational autoencoder <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\beta$</tex>-VAE). Through the dual-encoder system, it effectively disentangles the timbre and pitch, enabling the transfer of timbral characteristics across different musical notes and instruments while preserving the original pitch information. The performance of our approach, evaluated through metrics including Frechet Audio Distance (FAD), pitch accuracy, and Structural Similarity Index (SSIM), suggests its effectiveness for timbre conversions. This research offers potential applications in music production, audio synthesis, and sound design, paving the way for more adaptable and efficient timbre transfer solutions.