Robocskbench: Benchmarking Embodied Commonsense Capabilities of Large Language Models

Robots and intelligent assistants are increasingly performing tasks autonomously in household settings. While navigation-based tasks are straightforward, open-ended tasks require reasoning on the basis of commonsense knowledge. Towards fostering the development of systems that can use and reason on commonsense knowledge to tackle open-ended tasks, we propose RoboCSKBench, a natural language-based multitask benchmark to assess embodied commonsense knowledge capabilities of agents and systems interacting in dynamic household environments. Our benchmark combines various resources (e.g. knowledge graphs, manipulation benchmarks, crowdsourcing) to provide data for five different, commonly encountered household tasks: Tidy Up, Tool Usage, MetaReasoning, Table Setting and Procedural Knowledge. Each task comprises of data and evaluation metrics supporting the evaluation of systems inducing embodied commonsense knowledge. While the benchmark consists of five tasks at the time of writing, it can be extended with further tasks in the future. Building on the benchmark, we assess the capabilities of three state-of-the-art large language models on the various tasks of the benchmark. Our results indicate a diverse foundation, with model performance varying across different tasks, suggesting that no single model clearly outperforms the others. On the contrary, all models exhibit limitations, leaving room for further optimization and improvement.