<scp>AI</scp> Interpretation of <i>Ruijuhokogi</i> Headnotes: A Descriptive Preliminary Evaluation Using Microsoft Copilot

Interpreting classical texts of Japanese Traditional Medicine (JTM), also known as Kampo medicine, is challenging for non-specialists, including medical practitioners and healthcare professionals unfamiliar with the field. Additionally, a comprehension of Western medical knowledge is essential for understanding the diseases and pathological conditions described in these texts. Artificial intelligence (AI) is expected to support humans in carrying out such difficult tasks. Consequently, this study aimed to employ Microsoft Copilot, a generative AI [1], to map the prescription purposes described in the headnotes of Ruijuhokogi [2] to contemporary medical conditions and relevant clinical departments, and to evaluate the trends and validity of the AI-generated outputs. The text of the headnotes of Ruijuhokogi was manually entered into a database in Microsoft Excel, and the texts relating to the purpose of prescriptions were the subject of this study. Each target sentence was input into Copilot via copy and paste, accompanied by the standard query, “What is the Western medical name of the disease? Which specialist should I see?” The disease names and medical departments generated by Copilot were collected, and their appropriateness was assessed from a modern medical perspective by comparing them with pre-established expected disease names and medical departments to determine any significant deviations in each output. This study was conducted by a single evaluator, who interpreted the classical text using commentary sources such as Ruijuhokogikaisetsu (annotated by Fujihira) and investigated corresponding Western medical diagnoses by consulting online resources, including Medical Note and the MSD Manual to prepare expected disease names. This analysis encompassed a total of 327 sentences, covering nearly all texts related to the application of JTM. Additionally, inquiries were made to Copilot regarding potential improvements in its accuracy. In response to each query, Copilot generated approximately three disease names and some medical departments. Analysis of the Ruijuhokogi headnotes revealed that 68.2% (n = 223) of AI-generated diagnoses were fully appropriate (95% CI: 63.1%–73.3%), 28.1% (n = 92) were partially inappropriate (95% CI: 23.3%–33.0%), and 3.7% (n = 12) were completely inappropriate (95% CI: 1.6%–5.7%). For medical departments, 53.2% (n = 174) of outputs were within the expected range (95% CI: 47.8%–58.6%), while 46.8% (n = 153) required correction due to inappropriate or missing medical department names (95% CI: 41.4%–52.2%, calculated using the normal approximation method). Representative examples of inappropriate outputs and explanations for their occurrence are provided in the Supporting Information. The most frequently indicated conditions in the Ruijuhokogi headnotes were gastrointestinal diseases (Figure 1). Similarly, gastroenterology was the most frequently identified medical department corresponding to the written content. When queried on enhancing Copilot's reliability, it was noted that the AI searches for information using internal databases and external web searches, prioritizing data from trusted sources. Copilot is trained and continuously improved using anonymized data and feedback, with expert collaboration being crucial for model training. Academic papers and reputable websites are vital for external web searches. The headnotes of Ruijuhokogi were selected for this study due to the anticipated scarcity of information available on the Internet, thereby minimizing the likelihood of search results being influenced by pre-existing data. Copilot was used for its free accessibility, offering a practical option for users without paid AI services. In the future, it is anticipated that novices will primarily depend on diagnostic support AI software [3] and generative AI search services. It is essential to acknowledge that, by its nature, AI may occasionally produce incorrect responses. Concurrently, if AI is to be an educational and practical tool, there is a pressing need to enhance the accuracy of AI. AI technology is progressively being applied in traditional Chinese medicine. Zhou et al. indicated that the application of AI facilitates the extraction of valuable insights for clinical diagnosis and treatment from digitized ancient texts and expert experiments. However, they also emphasized that this process still necessitates professional clinical experience and the judgment of esteemed senior practitioners [4]. As reliance on AI for research and learning is projected to increase, it is crucial to ensure the dissemination and accumulation of accurate information. Failure to do so may result in AI generating theories that deviate from established JTM theories, potentially undermining the original theoretical framework. As limitations, this preliminary assessment relied on one evaluator, limiting precision. Future research should adopt designs that verify consistency and reproducibility. It is desirable to involve multiple JTM and related medical experts to calculate inter-rater agreement (e.g., κ, ICC) for Ruijuhokogi headings. In addition, evaluating outputs from multiple AI systems using the same set and comparing Ruijuhokogi with Shang Han Lun and among others could clarify applicability and reveal how source richness affects AI performance. For reference, the hallucination rate of GPT-4 turbo based Microsoft Copilot is 1.7% [5]. Finally, as an important point, given that AI derives answers by searching internal data and the Internet, it is incumbent upon us to ensure the accumulation of accurate data through feedback, publication of scholarly papers, and the creation of websites that accurately describe JTM theories. Yoshinobu Nakada evaluated the accuracy and error rates and drafted the manuscript. Masayoshi Koinuma performed the statistical analysis. The other authors critically reviewed and revised the manuscript. All authors read and approved the final version. Informed consent was obtained from Microsoft Copilot by inputting a query regarding its academic use. T. Nogami and M. Arai received lecture fees from Tsumura & Co. The authors declare no conflicts of interest regarding the reported facts or findings of this study. The datasets used and analyzed during the present study are available from the corresponding author upon reasonable request. Data S1: tkm270033-sup-0001-Supinfo.pptx. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.