A Small-Sample Sound Generation Methods Based on the Combination of Time and Frequency Domains

Contemporary neural network models for sound generation are often constrained by the lack of critical prior knowledge, which adversely affects the fidelity of the synthesized audio. Furthermore, these models typically produce intermediate feature maps that require an additional decoding step to reconstruct the corresponding audio signals, thereby introducing inefficiencies in the generation pipeline. To address these limitations, this study proposes an end-to-end model for speech generation. To tackle the challenges posed by long-duration sound sequences, the framework incorporates the Mamba model into its architecture, significantly improving computational efficiency and audio quality. Additionally, the model employs a novel loss function that integrates multimodal spectral and envelope features, facilitating joint optimization across temporal and spectral domains to enhance the naturalness and realism of the generated audio. Experimental evaluations demonstrate that the proposed approach achieves superior performance in synthesizing a diverse array of environmental sounds, surpassing state-of-the-art methods on both public datasets. This work represents a significant advancement in end-to-end sound generation, with broad implications for applications in virtual assistants, interactive gaming, and language learning technologies.