Generating Music With Retention Based Latent Transformer Models

This paper introduces a new approach of music generation by combining Contrastive Language-Audio Pre-training (CLAP) for text encoding, retention-based transformers for audio synthesis, and Vector Quantized Variational Autoencoders (VQVAE) to compress the audio. The quality of generated output improves and computation needs decline with comparison to a base transformer architecture. In this experiment, several model configurations have been tried, which dif- fer by the number of transformer blocks or hidden layer sizes. The configurations are rated by metrics derived from the Friedrich Audio Distance combined with training loss. The best performing model used 16 transformer blocks, along with a hidden dimension of 512, to achieve good performance while cutting training by a time of 10 times and inference by 20 times compared with the standard transformers.The MusicBench dataset was used, where both the training and testing employed the same dataset, using varied music samples accompanied by textual descriptions. This dataset offered a massive test of the model's ability to produce music in various styles and genres. The current study adds great value to the AI-generated music domain by presenting a strong architecture that will endure high-quality output while overcoming computational challenges most often cited as being associated with music generation models.