Design of Real-time Interactive Healthcare System for Acupoint Analysis Based on Augmented Reality

Acupressure is a valuable method for maintaining good health and also helps in the overall regulation of bodily functions and the promotion of physical fitness.In this paper, we propose a real-time interactive healthcare system for acupoint analysis based on augmented reality (AR).We utilized the edge computing of the Jetson Nano J1010 development board, and employed MediaPipe technology and an acupoint mapping algorithm to display acupoints on the screen.By using our proposed deep neural network (DNN)-based hand gesture recognition, we achieved AR-based interaction with acupoints.Users can view the locations of the corresponding acupoints on the body and the descriptions of the acupoints through gestures.Additionally, we developed a mobile app that enables interaction with the system via Bluetooth technology.Users can select the symptoms they wish to understand, and the system will display acupoints that can alleviate those symptoms.We also developed a bidirectional encoder representations from transformers (BERT)-based AI consultation, which utilizes the open-source conversational receive, appreciate, summarize, ask (RASA) natural language understanding (NLU) to determine the user's intents from input on the mobile app.When the intent is related to medical content, the information is passed to a BERT model for diagnosis analysis, determining possible diseases or symptoms and providing corresponding recommended acupoints for symptom relief.This aids users in making preliminary judgments about their own conditions.All AI diagnostic information is transmitted to the server side, where physicians can view user information through a website.By visualizing the data through charts, physicians can quickly understand the user's condition and further save medical resources.This acupoint-analysis-based healthcare system offers a safe, convenient, and efficient solution for health management, particularly valuable for individuals lacking experience and medical knowledge.We hope that by using this proposed system, users will feel like they have a virtual healthcare advisor at their disposal.