An end-to-end deep learning framework for structural damage assessment using semantic segmentation and point cloud analysis

The necessity for automated post-disaster building damage analysis using deep learning techniques arises from the critical need for rapid and accurate damage assessment following natural disasters. Traditional manual survey methods are time-consuming, labor-intensive, and potentially hazardous to ground personnel. To overcome these challenges, we propose a comprehensive pipeline that includes: (1) automated drone path planning to capture high-resolution 2D images of structures, (2) a novel Semantic Segmentation Hybrid model for segmentation of cracks and damages in structures, (3) 3D scene reconstruction from segmented multi-view 2D images using photogrammetry techniques, and (4) a novel 3D point cloud classification model built on PointNet for multi-class classification of damages on the 3D data to classify structural damage into three levels: No Damage, Heavy Damage, and Destruction. The proposed method achieves a recall of 76.2% and a mean IoU of 55.3% for segmentation tasks. For multi-level classification, the model demonstrates an overall accuracy of 78% and an F1 score of 97% for the Destruction level damage class. This framework significantly outperforms traditional manual assessment techniques by processing damaged data efficiently while maintaining high accuracy. • Novel CrackFusionNet Architecture CrackFusionNet introduces a pioneering hybrid approach to crack segmentation by combining U-Net with SE-ResNet50 encoder and Feature Pyramid Network (FPN) with DPN68 encoder. This fusion architecture leverages U-Net's symmetrical design for precise localization and FPN's multi-scale feature representation capabilities. Technically, the model concatenates outputs from both architectures along the channel dimension, processing them through a specialized fusion module with 3×3 and 1×1 convolutional layers, batch normalization, and ReLU activation to refine the combined feature maps. • Superior Performance Metrics of CrackFusionNet CrackFusionNet significantly outperforms state-of-the-art models in crack segmentation, achieving an accuracy of 0.9822, precision of 0.7379, recall of 0.7627, and F1 score of 0.7501. Notably, it surpasses DeepLabV3+ (0.9795 accuracy, 0.7321 F1 score) and standard FPN (0.9796 accuracy, 0.7298 F1 score). The model's maximum IoU of 0.9582 and average IoU of 0.5534 demonstrate its exceptional ability to accurately overlap predicted crack regions with ground truth. • Advanced Loss Function Implementation The segmentation model employs a sophisticated combined loss function that balances Dice Loss, Binary Cross-Entropy Loss, and Focal Loss with carefully tuned weights (1, 0.5, and 1.5 respectively). This strategic weighting addresses class imbalance issues inherent in crack detection, where crack pixels typically represent a minority class. The higher weight assigned to Focal Loss particularly enhances the model's ability to focus on hard-to-classify examples, significantly improving segmentation accuracy in challenging scenarios. • Innovative UPCLNet Architecture for Point Cloud Classification UPCLNet represents a significant advancement in 3D point cloud classification for structural damage assessment. Building on the PointNet backbone, it processes unordered sets of 3D points through input transformation, multi-layer perceptrons, and max pooling to aggregate features in a permutation-invariant manner. The architecture's unique approach enables it to effectively capture the spatial characteristics of building damage across three damage grades, processing 2048 points sampled from triangular meshes generated from photogrammetric data. • Unified Point Cloud Loss Function The paper introduces a groundbreaking Unified Point Cloud Loss (UPCL) function that integrates multiple loss components with optimized weighting coefficients: α =0.1 for Cross-Entropy Loss, β =0.5 for Chamfer Loss, γ =0.1 for Adversarial Loss, and δ =0.0001 for Regularization Loss. This unified approach ensures the model maintains geometric integrity while effectively learning classification boundaries, particularly excelling at distinguishing severe destruction cases with a remarkable F1 score of 0.97 compared to lower scores for no damage (0.65) and heavy damage (0.71). • Comprehensive End-to-End Pipeline for Damage Assessment The framework presents a first-of-its-kind integrated solution that combines semantic segmentation with multi-level 3D point cloud classification. This novel approach starts with UAV path planning for optimal multi-angle image capture, proceeds through crack segmentation with CrackFusionNet, utilizes photogrammetry for 3D reconstruction, and culminates in damage classification with UPCLNet. This comprehensive methodology addresses limitations of existing systems that typically focus on either segmentation or classification in isolation. • Automated UAV Path Planning Strategy The paper details an innovative drone path planning process that begins with an initial exploration flight to create a coarse model, followed by viewpoint planning based on completeness, precision, and semantic analysis. This methodology ensures optimal image capture from multiple angles with 70-80% overlap, generating 200-500 high-resolution images per structure to capture intricate damage details. The approach guarantees comprehensive data collection for subsequent processing stages, significantly enhancing the quality of damage assessment. • Photogrammetric 3D Reconstruction Techniques The research employs a sophisticated two-stage approach for 3D reconstruction that first generates point clouds using DBSCAN (Density-Based Spatial Clustering of Applications with Noise) for effective spatial data clustering, followed by CNN optimization to refine the mesh by learning surface smoothness and texture details. This methodology transforms 2D segmented images into precise 3D coordinates, creating a permanent, detailed record of structural damage that enables comprehensive analysis of damage extent and nature. • Real-World Performance Validation CrackFusionNet demonstrates exceptional performance on low-resolution, open-source UAV-captured images at various distances, accurately detecting cracks in real-world scenarios. The model successfully predicts major cracks from both long and medium-range UAV perspectives, outperforming human-annotated ground truth in some cases by identifying subtle defects overlooked in manual labeling. This confirms the model's practical applicability despite the challenges posed by varying image qualities and capture distances. • Computational Efficiency Trade-Offs While acknowledging the computational intensity of the hybrid architectures (particularly CrackFusionNet), the paper provides a candid analysis of the performance-efficiency trade-off. UPCLNet processes data efficiently with PointNet's architecture (capable of 1883 instances/second inference speed), making it suitable for real-time applications despite processing 2048 points per sample. The paper notes that future work will focus on optimizing these architectures for deployment on UAV onboard processors to enable real-time structural damage assessment in disaster scenarios.