Developing Multimodal Deep Learning Models for Technical Documents Representation and Retrieval

Existing approaches for information retrieval applied on technical documents have overlooked the potential of analyzing technical drawings in conjunction with textual data to enhance the accuracy of prior art searches. Integrating visual information with textual information to bridge the gap between visual perception and language understanding can significantly support the Engineering Design (ED) process. The ED process is a series of steps that engineers follow to find a solution to a problem. The steps include problem solving processes such as, for example, determining your objectives and constraints, prototyping, testing and evaluation. In fact, images provide synthetic representation of design artifact/process, and they emerge as the primary mode of communications among innovators, engineers, and designers throughout the ED phases. Pre-existing works did not perform tasks like image rotation, segmentation or specializing on a specific domain, increasing the complexity of the input given to the retrieval system and making more difficult to find problems if they arise. The proposed solution comprises a full dataset creation pipeline to polish the input and create descriptions aligning images with sentences in documents text. A model architecture definition, specialized for the given problem, is also provided. The results shows that performing retrieval with embeddings generated by a CLIP like model may not be enough and that more complex information retrieval pipelines are needed in order to handle technical documents.