Interactive English reading comprehension assistance based on vision transformers and natural language processing
English reading comprehension is a critical skill in modern education, often hindered by the limitations of existing text-only natural language processing (NLP) models, which fail to account for visual and structural elements such as layout and typography. To address this, we propose a multimodal framework integrating Vision Transformers (ViTs) and NLP techniques to enhance comprehension by combining visual and linguistic features through a cross-attention mechanism. The ViTs extract visual cues like document structure, spatial alignment, and emphasis from high-resolution optical imaging inputs, while the NLP modules process semantic and syntactic information. Optical Character Recognition (OCR) preprocessing further enhances text quality and provides bounding box information to assist ViTs in layout understanding, resulting in a unified understanding of text and visuals. Experimental evaluations on benchmark datasets, including SQuAD 2.0 and NewsQA, demonstrate significant performance improvements, achieving F1-score increases of 6.1% over text-only models and 17.3% over visual-only models. The system also proves highly adaptable to complex scenarios, such as ambiguous questions and visually emphasized content, while maintaining competitive computational efficiency with an average response time of 120 milliseconds per input. This research highlights the potential of multimodal approaches for AI-driven educational tools, paving the way for interactive and personalized learning experiences. Future work will focus on improving scalability, handling irregular layouts, and extending applications to multilingual and cross-domain contexts.
Authors
- Xiaohong Zhou
Institutions
- Wuhan Donghu University (CN)
Publication Details
- Published
- 2025-06-18
- DOI
- https://doi.org/10.1117/12.3073468
- Primary Topic
- Educational Technology and Pedagogy
- Type
- article
- Field-Weighted Citation Impact
- 0.00