LLM-Driven Automatic Reward Optimization for Ambient Intelligence Robots with Curriculum Interference Auto-Correction

Abstract Service robots in ambient intelligence (AmI) scenarios such as smart factories and home services must rapidly adapt to diverse manipulation tasks, yet traditional reinforcement learning (RL) reward design heavily relies on domain expertise, requiring days to weeks of manual tuning. Existing LLM-assisted reward design methods assume that generated rewards can be directly applied to training environments, overlooking potential incompatibility with built-in training mechanisms such as curriculum learning. To our knowledge, this is the first study to quantify the signal interference phenomenon between LLM-generated rewards and curriculum dynamic weight adjustments---experiments reveal that curriculum modules amplify penalty weights from an initial $-1\times10^{-4}$ to $-0.1$ (a factor of 1000$\times$), overwhelming the positive reward signals---and it proposes an automatic detection and correction mechanism based on feedback ratio anomaly detection. A unified dual-paradigm framework adapting to both ManagerBasedRL and DirectRL paradigms on the IsaacLab platform, along with a humanized interaction paradigm for non-expert users, are also developed. Systematic experiments on five robot tasks of varying difficulty, including ablation studies, 8-sample random search baselines, and multi-seed statistical tests, demonstrate: on Franka Lift the success rate increases from 0\% to 84\%$\pm$2\% and the mean reward from 0.90 to 84.41$\pm$5.50, where simply disabling the curriculum yields a roughly 5$\times$ increase in mean reward (0.90$\rightarrow$4.47)---a necessary prerequisite for LLM reward optimization to take effect---and combining it with the LLM-designed multi-stage reward structure multiplies the improvement by a further 18.9$\times$; under an equal-compute random-search comparison, LLM weight optimization achieves a 47\% improvement over the best random configuration. Mean-reward ratios depend on the reward definition and should be read together with the scale-invariant success rate. The method reduces reward design time from expert-days to hours, validated on the commercial LLM DeepSeek-V4-Pro.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-16
DOI
https://doi.org/10.21203/rs.3.rs-11453076/v1
Primary Topic
Reinforcement Learning in Robotics
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

LLM-Driven Automatic Reward Optimization for Ambient Intelligence Robots with Curriculum Interference Auto-Correction

Pengyun Bi, 张志东, Yongqiang He, Ruixian Ren
Research Square
Reinforcement Learning in Robotics
preprint

LLM-Driven Automatic Reward Optimization for Ambient Intelligence Robots with Curriculum Interference Auto-Correction

Pengyun Bi, 张志东, Yongqiang He, Ruixian Ren
preprint en

Abstract

Abstract Service robots in ambient intelligence (AmI) scenarios such as smart factories and home services must rapidly adapt to diverse manipulation tasks, yet traditional reinforcement learning (RL) reward design heavily relies on domain expertise, requiring days to weeks of manual tuning. Existing LLM-assisted reward design methods assume that generated rewards can be directly applied to training environments, overlooking potential incompatibility with built-in training mechanisms such as curriculum learning. To our knowledge, this is the first study to quantify the signal interference phenomenon between LLM-generated rewards and curriculum dynamic weight adjustments---experiments reveal that curriculum modules amplify penalty weights from an initial $-1\times10^{-4}$ to $-0.1$ (a factor of 1000$\times$), overwhelming the positive reward signals---and it proposes an automatic detection and correction mechanism based on feedback ratio anomaly detection. A unified dual-paradigm framework adapting to both ManagerBasedRL and DirectRL paradigms on the IsaacLab platform, along with a humanized interaction paradigm for non-expert users, are also developed. Systematic experiments on five robot tasks of varying difficulty, including ablation studies, 8-sample random search baselines, and multi-seed statistical tests, demonstrate: on Franka Lift the success rate increases from 0\% to 84\%$\pm$2\% and the mean reward from 0.90 to 84.41$\pm$5.50, where simply disabling the curriculum yields a roughly 5$\times$ increase in mean reward (0.90$\rightarrow$4.47)---a necessary prerequisite for LLM reward optimization to take effect---and combining it with the LLM-designed multi-stage reward structure multiplies the improvement by a further 18.9$\times$; under an equal-compute random-search comparison, LLM weight optimization achieves a 47\% improvement over the best random configuration. Mean-reward ratios depend on the reward definition and should be read together with the scale-invariant success rate. The method reduces reward design time from expert-days to hours, validated on the commercial LLM DeepSeek-V4-Pro.

Research Square
Shanxi Institute of Technology (CN)
Reinforcement Learning in Robotics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon