Diagnostics-LLaVA

The recent advancements in the area of Large language models (LLMs) has opened horizons for conversational assistant-based intelligent models capable of interpreting images, and providing textual response, also known as Visual language models (VLMs). These models can assist equipment operators and maintenance technicians in the complex Prognostics and Health Management (PHM) tasks such as diagnostics of faults, root cause analysis and repair recommendation. Significant open source contributions in the area of VLMs have been made. However, models trained on general data fail to perform well in complex tasks in specialized domains such as diagnostics and repair of industrial equipment. Therefore, in this paper we discuss our work on development of Diagnostics-LLaVA, a VLM suitable for interpreting images of specific industrial equipment and provide better response than existing open source models in PHM tasks such as fault diagnostics and repair recommendation. We introduce Diagnostics-LLaVA based on the architecture of LLaVA, and created one instance of the Diagnostics-LLaVA for the automotive repair domain, referred to as Automotive-LLaVA. We demonstrate that our proposed Automotive-LLaVA model performs better than the state-of-the-art open source visual language models such as mPlugOWL and LLaVA in both qualitative and quantitative experiments.