A Hybrid Multimodal Transformer-Based Approach for Text Extraction and Audio Generation from Images Using NLP

Abstract— A receipt or invoice is still one of the most common ways an everyday transaction gets put on paper, but pulling clean data out of a photograph of one is harder than it looks. Skewed lighting, odd angles, and cluttered backgrounds all chip away at how much a standard optical character recognition read can be trusted, and even once the words are out, a second problem remains: figuring out which word is the vendor name, which is the date, and which is the total. This paper describes a hybrid pipeline that pairs a deep-learning OCR engine with a transformer that reads text, position, and visual styling together rather than treating recognized words as a flat sequence. A rule-based fallback and a second OCR pass keep the system producing usable output even when the learned model cannot confidently label a field, and the resulting structured data is also converted into spoken audio, aimed at helping visually impaired users read printed documents they could not otherwise access unassisted. The complete pipeline was trained and evaluated on a public receipt dataset, with design choices checked against a text-only baseline and against manual testing on real-world receipts collected outside the training set.