EFR–GVA EvalKit: An Evaluation Framework for the Design and Development of Interactions between Emergency First Responders and Generative Voice Agents EICS011

Interactions between emergency first responders (EFRs) and Generative Voice Agents (GVAs) are shaping the future of communication in crisis management operations and training. Key issues are the associated security, privacy and ethical concerns of centralized, proprietary, remote GVAs especially in critical tasks. Remote GVAs also impact response times and require high power consumption infrastructure. These characteristics hinder the usability of GVAs since earlier voice assistants. However, open-source, local GVAs provide secure and efficient alternatives, but no standardized process exists to build GVAs in line with EFR’s requirements. This study proposes a unified evaluation framework, EFR-GVA EvalKit, with metrics of regulatory compliance, technical specifications and performance tests inline with EFRs’ applications requirements that prioritize transparency, data sovereignty and usability. We then used the EFR-GVA EvalKit framework to assess and compare the design and development of local GVAs on consumer grade devices with a frontier remote conversational agent in a triage training 3D scenario. A user study (N=19) with EFRs showed the capacity of local GVAs to meet EFRs’ expectations. Compared to existing approaches, our contribution is more comprehensive and engineered for reuse. EFR–GVA EvalKit is introduced in this paper as a new, compliance-gated evaluation method and the triage scenario is its first instantiation and validation, demonstrating how the framework can be replicated across devices and tasks.