A multi-dimensional concept for evaluating NLP-based robots in healthcare scenarios

Abstract Background: In the past decades, robots have transformed the healthcare sector by supporting clinical staff in various tasks. Applications of range from robot-supported surgical interventions and imaging, via healthcare logistics to cleaning and social robots. With the availability of natural language processing (NLP) methods based on large-language models (LLMs), the next generation of healthcare robots will in the next couple of years be able to communicate with humans in a completely new manner. Objective: To evaluate such NLPbased robots in various healthcare scenarios, adequate evaluation methods must be defined and implemented. Methods: For the evaluation of NLP-based robots a multi-dimensional framework is proposed, consisting of four dimensions and a living lab: D1: confidence, correctness and certifiability of LLMs for speech-based robots; D2: usability, acceptance and specifics of human-robot interaction (HRI); D3: the potentials to relieve clinical staff, and D4: the prospective technologyreflective analysis of HRI with respect to ethical, legal and social implications (ELSI) and its normative design. Additionally, LL: a living lab resp. a regulatory sandbox serves as a central hub for testing and validating future NLP-based robotic technologies under realistic conditions. Resume: Through this concept for evaluation, we intend to optimize the impact in the field of speech-based and no-code/low-code robotics.