Real-Time Avatar-Base Speech-to-Speech Conversational AI Tutor on AI PC

Large Language Models (LLMs) have become integral to daily life, powering popular applications like ChatGPT, Claude, and DALL-E. However, current solutions often fall short of consumer expectations for human-machine interaction (HMI). This paper presents an innovative HMI approach with both visual and audio capabilities: an Avatar-based Speech-to-Speech (S2S) Conversational AI Agent designed for educational applications, which scales as an AI Tutor. Our design employs a multimodal framework, integrating a series of AI components such as Noise Reduction, Automatic Speech Recognition (ASR), Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), Text-toSpeech (TTS), and Avatar modeling. This setup operates entirely on edge devices, like AI PCs, providing private and personalized learning content via RAG. Another objective of this paper is to introduce the End-to-End (E2E) solution usability scoring approach, called Composite Usability Score (CUS) to quantify the usability performance systematically in using the AI Tutor. With this CUS approach, it will bring up the usefulness of AI Tutor design in AI PC.