From image data to semantic insights: enhancing construction visual analytics through knowledge formalization and utilization

Digital transformation is increasingly demanded in Construction Management (CM) to boost operational efficiency. In this context, image data is gaining significant attention from practitioners due to its ease of deployment, intuitive nature, and ability to integrate seamlessly into workflows without disruption. Advances in computer vision algorithms have transformed image data from a passive documentation tool into a valuable source of actionable insights. Specifically, these algorithms enable image data to yield two types of information: low-level visual information, which focuses on identifying and categorizing objects and patterns within construction scenes, and high-level semantics, which integrate contextual information and reasoning to generate a deeper understanding of construction scenes. In particular, high-level semantics facilitate context-aware, reasoning-driven decision-making, reducing practitioners' cognitive burden and allowing them to focus on resolving project constraints, thereby improving overall CM efficiency. However, most existing research has not systematically investigated how to emphasize high-level semantics to enhance construction visual analytics, leaving a critical gap that this thesis aims to address. To enhance construction visual analytics, the first essential step is to establish a formal knowledge foundation that enables the structured representation of construction scene-related information. Such a foundation ensures interpretability for both human and machine agents, thereby effectively bridging the semantic gap between low-level visual information and high-level semantics. However, current knowledge representation efforts overlook a broader opportunity to systematically utilize construction scene-related knowledge, limiting their adaptability and practical applicability. To overcome this challenge, this thesis introduces the Construction Semantic Enrichment ontology (ConSE). ConSE integrates context-aware image content, high-level semantic concepts, and low-level visual information into a unified framework, systematically linking low-level visual information with high-level semantics. By doing so, ConSE provides a structured framework for developing semantic-enriched construction visual analytics systems. Evaluation results demonstrate that ConSE effectively supports key aspects of the development lifecycle, including resource selection, semantic enrichment, semantic annotation, and information retrieval, thereby laying the groundwork for context-aware construction visual analytics. With a structured knowledge foundation in place, it becomes possible to identify which low-level visual information is essential for inferring specific high-level semantics. This identification necessitates deploying targeted computer vision algorithms to accurately extract this information. Despite significant advancements in deep learning for low-level information extraction, applying these techniques in construction environments remains challenging due to the inherent complexity and variability of construction sites. To train robust low-level vision models, datasets must be able to capture the diversity and dynamic nature of construction environments. To address this challenge and automate dataset creation, this thesis proposes an ontology-guided text-to-image synthesis workflow. In this workflow, the ConSE aids in generating structured textual prompts for text-to-image models, guiding them to produce synthetic images that exhibit rich pattern diversity and visual variation while maintaining semantic consistency. Furthermore, the structured concepts in ConSE help mitigate hallucinations in generated images, improving their reliability. Experimental results demonstrate that this workflow effectively captures the dynamic nature of construction scenes, making it adaptable to diverse high-level semantic scenarios to generate synthetic images. Finally, these synthetic images, annotated with corresponding low-level information, contribute to creating large, diverse datasets for training low-level vision models with improved generalization capabilities in real-world construction environments. Once low-level visual information has been accurately extracted by robust low-level vision models, the final step is to aggregate this information into high-level semantics. Instead of directly mapping low-level visual information to high-level semantics through end-to-end learning approaches, reasoning-based approaches provide an advantage in terms of interpretability, which is essential for on-site decision-making. However, traditional reasoning-based approaches in the construction domain lack the flexibility to adapt to diverse and evolving construction environments, as they primarily rely on predefined rules and static knowledge representations. To address this limitation, this thesis explores the potential of Large Language Models (LLMs), which have demonstrated remarkable capabilities in pattern recognition and possess extensive world knowledge. However, since LLMs are pretrained on vast text corpora, applying their reasoning abilities to visual data from construction images requires bridging the gap between visual and textual representations. To achieve this, this thesis proposes an ontology-based prompting approach that leverages the structured knowledge foundation provided by ConSE. The approach first extracts relevant features from construction images using fundamental computer vision algorithms and maps them to predefined concepts within ConSE, creating a structured representation of visual data. It then organizes these structured representations alongside corresponding high-level semantics, serving as in-context examples for LLMs to learn and reason about construction-specific insights. Experimental results confirm the effectiveness of this approach in enabling LLMs to infer high-level semantics from real-world construction images. The ontological foundation and the context-agnostic prompting strategy employed in the approach suggest that this approach holds promise for broader engineering applications, offering a systematic framework for integrating low-level visual information with high-level semantic reasoning in construction visual analytics. This thesis introduces a comprehensive, knowledge-driven framework to advance construction visual analytics. By integrating formalized knowledge representation, ontology-guided dataset generation, and LLM-based reasoning, it lays a foundation for more interpretable, robust, and adaptable visual analytics in the construction domain. The proposed methodologies pave the way for future advancements in automated visual analysis and AI-driven decision support systems, driving the digital transformation of CM.