Multimodal Augmented Reality System for Real-Time Roof Type Recognition and Visualization on Mobile Devices

The utilization of augmented reality (AR) is becoming increasingly prevalent in the integration of virtual reality (VR) elements into the tangible reality of the physical world. It facilitates a more straightforward comprehension of the interconnections, interdependencies, and spatial context of data. Furthermore, the presentation of analyses and the combination of spatial data with annotated data are facilitated. This is particularly evident in the context of mobile applications, where the combination of real-world and virtual imagery facilitates enhances visualization. This paper presents a proposal for the development of a multimodal system that is capable of identifying roof types in real time and visualizing them in AR on mobile devices. The current approach to roof identification is based on data made available by public administrations in an open-source format, including orthophotos and building contours. Existing computer processing technologies have been employed to generate objects representing the shapes of building masses, and in particular, the shape of roofs, in three-dimensional (3D) space. The system integrates real-time data obtained from multiple sources and is based on a mobile application that enables the precise positioning and detection of the recipient’s viewing direction (pose estimation) in real time. The data were integrated and processed in a Docker container system, which ensured the scalability and security of the solution. The multimodality of the system is designed to enhance the user’s perception of the space and facilitate a more nuanced interpretation of its intricacies. In its present iteration, the system facilitates the extraction and classification/generalization of two categories of roof types (gable and other) from aerial imagery through the utilization of deep learning methodologies. The outcomes achieved suggest considerable promise for the advancement and deployment of the system in domains pertaining to architecture, urban planning, and civil engineering.