Multi-modal document classification in AEC asset management

The digitalization of asset management within the architecture, engineering and construction (AEC) sector is in need of effective methods for the automatic classification of documents. This study focuses on the development and evaluation of multimodal document classification models, utilizing visual, textual, and layout-related document information. We examine various state-of-the-art machine learning models and combine them through an iterative development process. The performance of these models is evaluated on two different AEC-document datasets. The results demonstrate that each of the modalities is useful in classifying the documents, as well as the integration of the different information types. This study contributes by applying AI techniques, specifically document classification in the AEC sector, setting the initial step to automating information extraction and processing for Intelligent Asset Management, and lastly, by combining and comparing multimodal state-of-the-art classification models on real-life datasets. • Multimodal AI enhances document classification in the AEC sector. • Combining text, visual, and layout features improves the accuracy of multimodal algorithms. • Multimodal AI supports automation for intelligent asset management in AEC.