Enhancing Daily Reports: Application of Multimodal GPT for Image Description Generation in Construction Site Quality Assurance

In recent years, cutting-edge language models like GPT, TAIDE, LLaMa, and VLM have exhibited remarkable prowess across diverse natural language tasks. This surge in artificial intelligence and computer vision technologies has revolutionized image analysis, offering a fresh and viable approach. Yet, the management and analysis of construction site images persist as formidable challenges in construction projects. While these images harbor valuable insights, conventional manual analysis methods are plagued by inefficiency and error susceptibility. Our study introduces an innovative multimodal approach named Daily Report for Construction Site GPT (DrCS-GPT), designed to navigate the intricacies of building construction sites. The study was initiated by compiling a diverse dataset encompassing visual images and reports from building construction sites in collaboration with CECI Engineering Consultants, Inc., Taiwan. In the data acquisition, the focus shifted towards refining our proposed model through pre-training and fine-tuning processes. This model exhibits proficiency in generating high-quality textual content while addressing common issues such as fragmented sentences and word repetition. The methodology entailed dataset assembly, model refinement, and integration with advanced linguistic tools to ensure precision and coherence in scientific communication. Our results underscore the model’s exceptional performance on 1600 images, with inference times of just 0.21s per task.