Benchmarking robots for everyday environments: from lab experiments to real-world operations

This study presents an interdisciplinary process for benchmarking service robots in public everyday environments, addressing the gap between laboratory metrics and real-world operation at technology readiness levels 6–7. Three robots were evaluated in three use cases: outdoor park cleaning, pedestrian underpass cleaning, and interactive assistance in a public library. Over three years (2023–2025), seven benchmarking events (a consensus workshop and six on-site evaluations, two per use case) assessed the robots in increasingly realistic settings, from controlled conditions to public operation. An interdisciplinary panel of experts in robotics, human–robot interaction, safety, and economic viability consolidated use-case-specific metrics into a shared evaluation concept refined across three phases. Rather than ranking the robots, the evaluation examined how task fulfillment, interaction quality, safety, and economic viability interact in the transition from laboratory prototype to operational system. Direct quantitative comparability across the heterogeneous use cases remained limited, characteristic of benchmarking at this maturity level. The process yielded cross-case insights: increasing environmental realism introduces challenges that uniform metrics capture only in part; the interaction-quality guideline works best as a modular instrument adapted to each use case; safety was assessed by occupational-safety and product-certification experts who adapted industrial test procedures for public settings; and economic viability proved a cross-cutting dimension. We conclude that benchmarking at higher technology readiness levels is well served by a shared procedure for selecting and documenting metrics across these four categories, with safety as mandatory, rather than by a fixed set of metrics: generalizability lies in the procedure rather than in identical metrics.

Authors

Institutions

Publication Details

Journal
Frontiers in Robotics and AI
Published
2026-11-13
DOI
https://doi.org/10.3389/frobt.2026.1913018
Primary Topic
Robotics and Automated Systems
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Benchmarking robots for everyday environments: from lab experiments to real-world operations

Marlene Wessels, Raphael Memmesheimer, Sven Behnke, Simone Roth et al.
Frontiers in Robotics and AI
Robotics and Automated Systems
article

Benchmarking robots for everyday environments: from lab experiments to real-world operations

Marlene Wessels, Raphael Memmesheimer, Sven Behnke, Simone Roth, Alexander Arntz, Tristan Schnell, Martin Gersch, Wilhelm Stork, Arne Roennau, Paul Schweidler, Benedikt Pulver, Carolin Straßmann, Martina Overbeck, Björn Kral, Johannes Kraus, Medina Klicic, Sabine Bellmann, Dominik Beyer, Anna-Maria Meer, Sven Schneider, Christoph Zimmermann, Jan Zimmermann
article en

Abstract

This study presents an interdisciplinary process for benchmarking service robots in public everyday environments, addressing the gap between laboratory metrics and real-world operation at technology readiness levels 6–7. Three robots were evaluated in three use cases: outdoor park cleaning, pedestrian underpass cleaning, and interactive assistance in a public library. Over three years (2023–2025), seven benchmarking events (a consensus workshop and six on-site evaluations, two per use case) assessed the robots in increasingly realistic settings, from controlled conditions to public operation. An interdisciplinary panel of experts in robotics, human–robot interaction, safety, and economic viability consolidated use-case-specific metrics into a shared evaluation concept refined across three phases. Rather than ranking the robots, the evaluation examined how task fulfillment, interaction quality, safety, and economic viability interact in the transition from laboratory prototype to operational system. Direct quantitative comparability across the heterogeneous use cases remained limited, characteristic of benchmarking at this maturity level. The process yielded cross-case insights: increasing environmental realism introduces challenges that uniform metrics capture only in part; the interaction-quality guideline works best as a modular instrument adapted to each use case; safety was assessed by occupational-safety and product-certification experts who adapted industrial test procedures for public settings; and economic viability proved a cross-cutting dimension. We conclude that benchmarking at higher technology readiness levels is well served by a shared procedure for selecting and documenting metrics across these four categories, with safety as mandatory, rather than by a fixed set of metrics: generalizability lies in the procedure rather than in identical metrics.

Frontiers in Robotics and AIVol. 13
Karlsruhe Institute of Technology (DE), University of Bonn (DE), FZI Research Center for Information Technology (DE), Johannes Gutenberg University Mainz (DE), Machine Intelligence Research Institute (US), Fraunhofer Institute for Manufacturing Engineering and Automation (DE), TÜV SÜD (Germany) (DE), Ruhr West University of Applied Sciences (DE), Institut für Arbeitsschutz der Deutschen Gesetzlichen Unfallversicherung (DE), HFC Human-Factors-Consult GmbH (Germany) (DE), Freie Universität Berlin (DE)
Bundesministerium für Bildung und Forschung
Sustainable cities and communities
Robotics and Automated Systems
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon