Diffcraft: A Modular Differentiable Framework for Music Synthesis with Timbre Transfer

This paper presents a novel autoencoder-based framework for music synthesis and timbre transfer, addressing limitations of traditional Differentiable Digital Signal Processing (DDSP). By integrating a Variational Autoencoder (VAE) based Audio Encoder with a Mamba-enhanced DDSP Decoder, the framework extracts fundamental frequency (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$F_{0}$</tex>) and loudness (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$L$</tex>) directly from waveforms, eliminating reliance on pre-extracted features. The use of Piano Complex Cepstral Coefficients (PCCC) with VAE allowed learning a smooth and rich feature space, enabling flexible audio manipulation. Bidirectional Mamba demonstrated improved performance over the baseline DDSP model. The system achieves high-quality audio generation and efficient timbre transfer. The proposed framework is versatile, offering applications in music transcription, timbre transfer, and multiinstrument synthesis.