Deep learning models for solving the problem of comparing unstructured textual information

This study is devoted to the application of modern deep learning methods, such as universal multilingual sentence vector representations, for analyzing and comparing the similarity of unstructured text documents. The paper develops an algorithm for determining the similarity of multilingual texts using deep neural networks. The authors propose a supervision-free method for fast analysis and comparison of complex documents on similar topics written in different countries and in different languages. The presented methodology involves pre-processing documents, reformatting them for comparison, and applying deep learning methods to detect textual similarities. The algorithm is tested on the example of a comparative analysis of the similarity of the texts of the Tax Code of Ukraine and the German fiscal code. The Ukrainian fiscal code, presented in HTML format, was processed using the library for analyzing HTML documents and regular expressions, while the German fiscal code in PDF format required special analysis of unstructured content. In the course of the study, to improve the accuracy and overcome the identified limitations of universal multilingual vector representations, the Ukrainian document was translated into English using the Google Translate API. Solving the given task requires deep pre-processing of the data, which was carried out in this study. The article provides a step- by-step algorithm of the proposed methodology, which leads to the formation of a correlation table, which determines the similarity between similar multilingual text documents. Additionally, the authors support every step with their reasoning about how to interpret intermediate results and understand if the method should be adjusted to the need or the problems of the specific documents, like applying machine translation or advanced pre-processing. The authors propose an algorithm for applying this method to the fiscal codes of European Union member states or other countries. In this research, we identified and described the limitations of this approach, demonstrated the verification of intermediate results, and supplemented the method with additional capabilities to increase its reliability. Recommendations are given on the possibilities of applying the proposed methodology, in particular, the development of algorithms for standardizing certain documents of the European Union.