Design and Research of Virtual Human Based on Deep Learning and Paddle Model

This paper investigates the application of deep learning and Paddle models in the field of virtual humans, developing a design method for virtual humans based on deep learning. With the rapid development of artificial intelligence technology, the technology of virtual personal assistants has also made significant breakthroughs, evolving towards more intelligent and realistic directions. The increasing demands for virtual assistants indicate the need for a novel design approach for virtual humans. This study first implemented intelligent Q&A through ChatGLM, then utilized the PP-TTS, a streaming speech synthesis system researched by PaddleSpeech, to achieve streaming speech synthesis. Post-processing of the generated voice was conducted using So-vits-svc to enhance the similarity between the cloned voice and the original. Secondly, the PaddleGAN's Wav2Lip model was employed to match the new video with the new speech for lip synchronization. Subsequently, a facial expression transfer model First Order Motion (FOM), which is based on Generative Adversarial Networks (GAN) and Optical Flow Estimation, was applied to transfer expressions from a speaking video to the corresponding still images of the person. Finally, the developed virtual human assistant was deployed on a local server for specific experiments, which verified the feasibility and effectiveness of the proposed solution and provided a solid foundation for subsequent system design and tuning.