Integrating Machine Learning & Deep Learning for Image Captioning and Text Recognition in Mobile Apps Development

Deep learning methods are increasingly being applied to mobile applications for word recognition and image captioning, offering innovative solutions to complex tasks. In this study, models such as VGG19, ResNet101, and InceptionV3 Hybrid models, functioning as encoders in conjunction with LSTM decoders, are leveraged to generate natural language descriptions of images using the Flickr8k dataset. Furthermore, various machine learning techniques—such as Support Vector Machines (SVM), Random Forest, K-Nearest Neighbors (KNN), Logistic Regression, Naive Bayes, and Decision Trees—are evaluated for Bangla text categorization. Deep learning models, including CNNs, hybrid models, and VGG19, are applied for classifying English text and non-text images. A unique hybrid model combining CNN, VGG19, and LSTM is introduced to differentiate between text and non-text images in the English dataset. For nontext images, vision-based frameworks are employed to generate image captions. Moreover, two mobile applications developed using Android Studio are presented: the first enables live text scanning with text-to-speech conversion, while the second supports OCR-based text recognition from gallery images, providing voice outputs in both Bangla and English. Additionally, the research extends to handwritten word recognition through CNNs, RNNs, and Connectionist Temporal Classification (CTC) models, alongside CAPTCHA image recognition utilizing a combination of CNNs, OCR, and CTC techniques.