Steering the Future: A Catalog of Failures in Deep Learning-Enabled Robotic Navigation Systems

Failure catalogs have proven to be key instruments driving the evolution and assessment of program analysis techniques. However, such infrastructure does not support the development of techniques for the large number of emerging robotic systems. Developing such a catalog is costly and challenging because it requires access to the full physical system and the presentation of a diverse set of failures. We have started to tackle this challenge, building Defects4DeepNav, a growing catalog of over 100 failures from a commercial open source robot operating in the real world navigated by a learned component, with a diverse set of failures arising from 30 navigation components and growing. This paper introduces Defects4DeepNav, including a diverse set of failures, full sensor data for failing and non-failing behavior, tools to analyze the data, and illustrations of its potential use cases and extensions.