From questions to functions - identifying ISO 7010 safety standards pictograms with AI vision

Abstract The International Organization for Standardization (ISO) 7010 standard is a critical component in maintaining safety and health in various environments, including workplaces, public areas and where heavy machinery is operated. ISO 7010 prescribes safety signs across standard images for the purposes of accident prevention and hazardous protection, providing a universal language of safety, that can be understood by everyone, regardless of their nationality or native language. These symbols are unique, having only one symbol for each meaning to avoid confusion across industries. In this context, this work proposes a solution to train, evaluate and deploy an AI image detection model that is able to identify such signs in an image (frame), and paired with natural language processing (NLP) through the use of proprietary Large Language Models (LLM), be able to identify in which location of the document such signs are present. For that, the framework uses cutting-edge LLM technologies as Function Tools and RAG (Retrieval Augmented Generation) practices. Results obtained in this framework accurately retrieves isolated pictograms from semantic queries, but faces reduced classification accuracy when images contain adjacent text. This work can also bring other potential benefits for places using ISO 7010 standards, such as enhanced safety compliance, real-time monitoring, accessibility and audit.