Mixed Text and Formula Recognition Using ResNet and Transformer

Recent advancements in deep learning have enabled the recognition of images, objects, observations, texts, and other complex structures with increasing accuracy. In recent years, scene text recognition has stimulated a lot of researchers in the computer vision community. However, it still needs improvement due to the poor performance of existing scene recognition algorithms. This paper aims to combine deep learning research to recognize texts and formulas in images and convert them into LaTeX format. More accurately and efficiently recognizing and converting images containing mixed text and mathematical formulas can improve the quality of the generated LaTeX code. In terms of the model, this paper utilizes ResNet as an Encoder for feature extraction and Transformer as a Decoder for text generation, improving the accuracy of the generated text. Two types of datasets were used in the process: pure formulas and a mixture of text and formulas. This paper concludes that the effects on both datasets are good, with CERs of less than 0.05 and a loss of only 0.10. Besides, the accuracy of pure formula recognition is higher than that of mixed types, possibly because there is more data on pure formulas and it is difficult to accurately extract text features in mixed types. This method can automatically parse and generate content containing mathematical formulas, promoting the development of the education industry and improving the efficiency and quality of digital content generation.