Hierarchical command and control for simulated agents through feudal proximal policy optimization

Training simulations require synthetic forces capable of intelligent, adaptive behavior, but currently rely on brittle, scripted, rule-based approaches or on human operators. Multi-agent deep reinforcement learning offers a promising alternative, but existing flat architectures lack the structure needed for top-down task allocation via explicit command structures. This study introduces Feudal Leader Independent Proximal Policy Optimization, a feudal multi-agent hierarchy comprising an independent leader and a follower model trained on a joint task. For simulation environments that model command hierarchies, Feudal Leader Independent Proximal Policy Optimization embeds structured task allocation directly into the learning architecture. Feudal Leader Independent Proximal Policy Optimization is evaluated on the Bitstring Coordination Problem, a parameterized optimization testbed that enables controlled analysis of sample efficiency and scalability, and on SMAClite, a higher-fidelity tactical multi-agent simulation. In the Bitstring Coordination Problem, Feudal Leader Independent Proximal Policy Optimization’s task performance matches or exceeds non-hierarchical baselines within 62.5% of the total training budget, with this advantage holding as teams scale from 3 to 9. Ablation analysis shows that both hierarchical structure and compliance reward are necessary for these gains. Under sparse reward at scale, all flat baselines fail, while Feudal Leader Independent Proximal Policy Optimization remains successful. In SMAClite, Feudal Leader Independent Proximal Policy Optimization achieves more victories than baseline approaches. These findings position feudal hierarchies as a viable foundation for fully autonomous forces in next-generation training simulations.

Authors

Institutions

Publication Details

Journal
SIMULATION
Published
2026-11-16
DOI
https://doi.org/10.1177/00375497261498851
Primary Topic
Reinforcement Learning in Robotics
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Hierarchical command and control for simulated agents through feudal proximal policy optimization

Sean C. Mondesire, Austin Starken
SIMULATION
Reinforcement Learning in Robotics
article

Hierarchical command and control for simulated agents through feudal proximal policy optimization

Sean C. Mondesire, Austin Starken
article en

Abstract

Training simulations require synthetic forces capable of intelligent, adaptive behavior, but currently rely on brittle, scripted, rule-based approaches or on human operators. Multi-agent deep reinforcement learning offers a promising alternative, but existing flat architectures lack the structure needed for top-down task allocation via explicit command structures. This study introduces Feudal Leader Independent Proximal Policy Optimization, a feudal multi-agent hierarchy comprising an independent leader and a follower model trained on a joint task. For simulation environments that model command hierarchies, Feudal Leader Independent Proximal Policy Optimization embeds structured task allocation directly into the learning architecture. Feudal Leader Independent Proximal Policy Optimization is evaluated on the Bitstring Coordination Problem, a parameterized optimization testbed that enables controlled analysis of sample efficiency and scalability, and on SMAClite, a higher-fidelity tactical multi-agent simulation. In the Bitstring Coordination Problem, Feudal Leader Independent Proximal Policy Optimization’s task performance matches or exceeds non-hierarchical baselines within 62.5% of the total training budget, with this advantage holding as teams scale from 3 to 9. Ablation analysis shows that both hierarchical structure and compliance reward are necessary for these gains. Under sparse reward at scale, all flat baselines fail, while Feudal Leader Independent Proximal Policy Optimization remains successful. In SMAClite, Feudal Leader Independent Proximal Policy Optimization achieves more victories than baseline approaches. These findings position feudal hierarchies as a viable foundation for fully autonomous forces in next-generation training simulations.

SIMULATION
University of Central Florida (US)
Reinforcement Learning in Robotics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon