Hybrid Visual Perception and LLM Framework for Service Robot Planning and Navigation

Abstract In the last decade, service robots have gained significant attention for their role in supporting humans across various applications. To operate effectively and adaptively, service robots require environmental perception, natural-language understanding, and planning and navigation capabilities. However, traditional planning frameworks struggle to provide the required flexibility. In contrast, end-to-end multimodal systems based on vision-language models (VLMs) require substantial computational resources, making them unsuitable for deployment on embedded robotic platforms. This paper proposes a hybrid framework that combines fast, deep-learning-based visual perception with the reasoning capabilities of a large language model (LLM) to enhance task comprehension, planning, and navigation for service robots. By decoupling vision from language reasoning, we avoid the complexity of a unified vision-language model, allowing the proposed framework to be deployed on a lightweight embedded platform with substantially lower latency than the VLM baseline. Experimental results show a 96\% task completion rate in structured mission tests and 84\% overall success in hybrid perception tasks. Moreover, the proposed approach achieves 3× to 20× faster inference than VLM-based baselines under both API-based and local deployment settings. Real-robot deployment confirmed the successful transfer of the full perception-to-action pipeline to the physical platform.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-12
DOI
https://doi.org/10.21203/rs.3.rs-11370496/v1
Primary Topic
Robot Manipulation and Learning
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Hybrid Visual Perception and LLM Framework for Service Robot Planning and Navigation

Dino Accoto, Igi Ardiyanto, Ahmad Ataka, Jun Miura et al.
Research Square
Robot Manipulation and Learning
preprint

Hybrid Visual Perception and LLM Framework for Service Robot Planning and Navigation

Dino Accoto, Igi Ardiyanto, Ahmad Ataka, Jun Miura, Yanayir Rifai, Burhanudin Burhanudin, Yusuf Badriawan
preprint en

Abstract

Abstract In the last decade, service robots have gained significant attention for their role in supporting humans across various applications. To operate effectively and adaptively, service robots require environmental perception, natural-language understanding, and planning and navigation capabilities. However, traditional planning frameworks struggle to provide the required flexibility. In contrast, end-to-end multimodal systems based on vision-language models (VLMs) require substantial computational resources, making them unsuitable for deployment on embedded robotic platforms. This paper proposes a hybrid framework that combines fast, deep-learning-based visual perception with the reasoning capabilities of a large language model (LLM) to enhance task comprehension, planning, and navigation for service robots. By decoupling vision from language reasoning, we avoid the complexity of a unified vision-language model, allowing the proposed framework to be deployed on a lightweight embedded platform with substantially lower latency than the VLM baseline. Experimental results show a 96\% task completion rate in structured mission tests and 84\% overall success in hybrid perception tasks. Moreover, the proposed approach achieves 3× to 20× faster inference than VLM-based baselines under both API-based and local deployment settings. Real-robot deployment confirmed the successful transfer of the full perception-to-action pipeline to the physical platform.

Research Square
Toyohashi University of Technology (JP), Universitas Gadjah Mada (ID), Flanders Make (Belgium) (BE), KU Leuven (BE)
Universitas Gadjah Mada
Robot Manipulation and Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon