A Transformer Substation Defect Detection Method Based on Multimodal Knowledge Graph

Most traditional knowledge representation learning methods only focus on structured information in triples, and cannot effectively utilize additional information such as entity images, relationship paths, and text descriptions to learn knowledge representation or only integrate one type of additional information. Therefore, a multimodal knowledge graph embedding method combining entity description and image is proposed, which can provide more comprehensive external information through mutual enhancement of text and image to compensate for the shortcomings of incomplete individual information sources in knowledge representation learning. Firstly, entity description and image modeling are performed to obtain text and image representations of entities, which are used as supplements to the structural representation in TransE. Finally, a unified spatial representation of knowledge graph and text and image is achieved through joint training of three entity representations, improving the accuracy of entity and relationship prediction. In order to improve the accuracy of substation equipment defect detection using a small number of samples, this paper proposes a defect detection method based on multimodal knowledge graph. The experimental results show that this method can fully utilize non iconic images for comparative learning, and improve the accuracy of substation equipment defect detection under the condition of only labeling a small number of samples, such as meter dial damage, hanging suspended solids, bird nests, etc. The mAP on five types of defect detection tasks, including silicone discoloration and abnormal door closure, reached 83.84%.