Interactive English reading comprehension assistance based on vision transformers and natural language processing

English reading comprehension is a critical skill in modern education, often hindered by the limitations of existing text-only natural language processing (NLP) models, which fail to account for visual and structural elements such as layout and typography. To address this, we propose a multimodal framework integrating Vision Transformers (ViTs) and NLP techniques to enhance comprehension by combining visual and linguistic features through a cross-attention mechanism. The ViTs extract visual cues like document structure, spatial alignment, and emphasis from high-resolution optical imaging inputs, while the NLP modules process semantic and syntactic information. Optical Character Recognition (OCR) preprocessing further enhances text quality and provides bounding box information to assist ViTs in layout understanding, resulting in a unified understanding of text and visuals. Experimental evaluations on benchmark datasets, including SQuAD 2.0 and NewsQA, demonstrate significant performance improvements, achieving F1-score increases of 6.1% over text-only models and 17.3% over visual-only models. The system also proves highly adaptable to complex scenarios, such as ambiguous questions and visually emphasized content, while maintaining competitive computational efficiency with an average response time of 120 milliseconds per input. This research highlights the potential of multimodal approaches for AI-driven educational tools, paving the way for interactive and personalized learning experiences. Future work will focus on improving scalability, handling irregular layouts, and extending applications to multilingual and cross-domain contexts.

Authors

Institutions

Publication Details

Published
2025-06-18
DOI
https://doi.org/10.1117/12.3073468
Primary Topic
Educational Technology and Pedagogy
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Interactive English reading comprehension assistance based on vision transformers and natural language processing

Xiaohong Zhou
Educational Technology and Pedagogy
article

Interactive English reading comprehension assistance based on vision transformers and natural language processing

Xiaohong Zhou
article en

Abstract

English reading comprehension is a critical skill in modern education, often hindered by the limitations of existing text-only natural language processing (NLP) models, which fail to account for visual and structural elements such as layout and typography. To address this, we propose a multimodal framework integrating Vision Transformers (ViTs) and NLP techniques to enhance comprehension by combining visual and linguistic features through a cross-attention mechanism. The ViTs extract visual cues like document structure, spatial alignment, and emphasis from high-resolution optical imaging inputs, while the NLP modules process semantic and syntactic information. Optical Character Recognition (OCR) preprocessing further enhances text quality and provides bounding box information to assist ViTs in layout understanding, resulting in a unified understanding of text and visuals. Experimental evaluations on benchmark datasets, including SQuAD 2.0 and NewsQA, demonstrate significant performance improvements, achieving F1-score increases of 6.1% over text-only models and 17.3% over visual-only models. The system also proves highly adaptable to complex scenarios, such as ambiguous questions and visually emphasized content, while maintaining competitive computational efficiency with an average response time of 120 milliseconds per input. This research highlights the potential of multimodal approaches for AI-driven educational tools, paving the way for interactive and personalized learning experiences. Future work will focus on improving scalability, handling irregular layouts, and extending applications to multilingual and cross-domain contexts.

Wuhan Donghu University (CN)
Quality Education
Openalex Percentile: Top 93%
Educational Technology and Pedagogy
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.