Principles and Guidelines for Evaluating Social Robot Navigation Algorithms

A major challenge to deploying robots widely is navigation in human-populated environments, commonly referred to as social robot navigation . While the field of social navigation has advanced tremendously in recent years, the fair evaluation of algorithms that tackle social navigation remains hard because it involves not just robotic agents moving in static environments but also dynamic human agents and their perceptions of the appropriateness of robot behavior. In contrast, clear, repeatable, and accessible benchmarks have accelerated progress in fields like computer vision, natural language processing and traditional robot navigation by enabling researchers to fairly compare algorithms, revealing limitations of existing solutions and illuminating promising new directions. We believe the same approach can benefit social navigation. In this article, we pave the road toward common, widely accessible, and repeatable benchmarking criteria to evaluate social robot navigation. Our contributions include (a) a definition of a socially navigating robot as one that respects the principles of safety, comfort, legibility, politeness, social competency, agent understanding, proactivity, and responsiveness to context, (b) guidelines for the use of metrics, development of scenarios, benchmarks, datasets, and simulators to evaluate social navigation, and (c) a design of a social navigation metrics framework to make it easier to compare results from different simulators, robots, and datasets.

Authors

Institutions

Publication Details

Journal
ACM Transactions on Human-Robot Interaction
Published
2024-12-27
DOI
https://doi.org/10.1145/3700599
Citations
53
Primary Topic
Social Robot Interaction and HRI
Type
article
Field-Weighted Citation Impact
33.47
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Principles and Guidelines for Evaluating Social Robot Navigation Algorithms

Peter Stone, Joydeep Biswas, Peng Xu, Peter Trautman et al.
53 citations
ACM Transactions on Human-Robot Interaction
Social Robot Interaction and HRI
33.47
article

Principles and Guidelines for Evaluating Social Robot Navigation Algorithms

Peter Stone, Joydeep Biswas, Peng Xu, Peter Trautman, Claudia Pérez-D’Arpino, Nathan Tsoi, Jonathan P. How, Rohan Chandra, Xuesu Xiao, Sören Pirk, Abhijat Biswas, Reuth Mirsky, Haresh Karnan, Marynel Vázquez, Anthony Francis, Phani Teja Singamaneni, Aniket Bera, Rachid Alami, Alexander Toshev, Michael Everett, Sehoon Ha, Roberto Martín-Martín, Ada V. Taylor, Alexandre Alahi, Tsang-Wei Edward Lee, Luis J. Manso, Justin Hart, Fei Xia, Chengshu Li, Naoki Yokoyama, Hao-Tien Lewis Chiang
article en
53 citations

Abstract

A major challenge to deploying robots widely is navigation in human-populated environments, commonly referred to as social robot navigation . While the field of social navigation has advanced tremendously in recent years, the fair evaluation of algorithms that tackle social navigation remains hard because it involves not just robotic agents moving in static environments but also dynamic human agents and their perceptions of the appropriateness of robot behavior. In contrast, clear, repeatable, and accessible benchmarks have accelerated progress in fields like computer vision, natural language processing and traditional robot navigation by enabling researchers to fairly compare algorithms, revealing limitations of existing solutions and illuminating promising new directions. We believe the same approach can benefit social navigation. In this article, we pave the road toward common, widely accessible, and repeatable benchmarking criteria to evaluate social robot navigation. Our contributions include (a) a definition of a socially navigating robot as one that respects the principles of safety, comfort, legibility, politeness, social competency, agent understanding, proactivity, and responsiveness to context, (b) guidelines for the use of metrics, development of scenarios, benchmarks, datasets, and simulators to evaluate social navigation, and (c) a design of a social navigation metrics framework to make it easier to compare results from different simulators, robots, and datasets.

ACM Transactions on Human-Robot InteractionVol. 14(2)
Honda (Japan) (JP), Northeastern University (US), Google (United States) (US), Centre National de la Recherche Scientifique (FR), Georgia Institute of Technology (US), Bar-Ilan University (IL), George Mason University (US), Aston University (GB), Université Fédérale de Toulouse Midi-Pyrénées (FR), Laboratoire d'Analyse et d'Architecture des Systèmes (FR), Purdue University West Lafayette (US), Christian-Albrechts-Universität zu Kiel (DE), Yale University (US), Simpson Strong-Tie (United States) (US), Nvidia (United States) (US), Honda (United States) (US), Apple (United States) (US), École Polytechnique Fédérale de Lausanne (CH), University of Virginia (US), Massachusetts Institute of Technology (US), Carnegie Mellon University (US), The University of Texas at Austin (US), Stanford University (US)
Reduced inequalities
Openalex Percentile: Top 0%
Social Robot Interaction and HRI
33.47
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.