A Modular AI-Driven Multimodal Framework for Real-Time Classroom Interpretation for Deaf Students

Abstract Despite significant advances in inclusive education, real-time classroom communication remains a major challenge for students with profound hearing impairment, particularly in science, technology, engineering, and mathematics (STEM) learning environments where continuous verbal interaction and concept-oriented instruction are essential. Existing assistive technologies, including speech-to-text captioning, frequency modulation (FM) systems, and human sign language interpreters, either provide limited contextual understanding, lack scalability, or are unavailable in many educational institutions. This paper presents a modular artificial intelligence (AI)-driven multimodal framework for real-time classroom interpretation designed to enhance accessibility for deaf learners. The proposed framework integrates Automatic Speech Recognition (ASR), educational semantic processing using Large Language Models (LLMs), Indian Sign Language (ISL) gloss generation, intelligent sign retrieval, and avatar-assisted visual communication within a unified architecture. In addition, the framework incorporates a systematic dataset engineering pipeline comprising vocabulary organization, automated metadata generation, annotation management, and gloss-based indexing to facilitate efficient retrieval and future scalability. Unlike conventional systems that primarily focus on isolated speech transcription or sign language translation, the proposed architecture emphasizes modularity, educational context awareness, extensibility, and deployment readiness for real classroom environments. The paper details the overall system architecture, workflow, and implementation methodology while highlighting the integration of multiple AI components into a cohesive assistive learning platform. The proposed framework establishes a scalable technological foundation for developing next-generation intelligent classroom interpretation systems and contributes toward advancing accessible, inclusive, and AI-enabled education for deaf students.