Experimental Analysis on Stable Diffusion Model for Speech-to-Image Transliteration

The transliteration of spoken description to visual representations is becoming more important in various fields such as education, communication etc. This work suggests an innovative method for generating high-quality images from speech inputs by combing the OpenAI's Whisper model for Speech Recognition and Stable Diffusion model for image synthesis. The also evaluates how the varying values of the parameters affect the quality of the generated images. The data of self-recorded audio files from educational content to descriptive content served as the dataset for this work. The objective of the work is to optimize the image generation process by tuning the parameters such as Guidance Scale, Inference Steps, eta(n), Top k, Top p and also to enhance the quality of images. The results showed variance with the generated images with default parameter values and the images after hyper parameter tuning. The outcome of the model for default and tuned parameters are compared in terms of Guidance Scale, Inference Steps, eta(n), Top k, Top p. The factors that are taken in analysis are, similar images in same content, time taken to generate images, and change in the quality of image. The audio used as dataset are self-recorded, which contains the content such as education content and stories. The experimental analysis supports to enhance the difficulties with the hyperparameter optimization and transcribing accuracy can be addressed to enhance the system for more complex use cases.