Modular Approach for Controlling Multi-Agent Systems with Natural Language

The integration of natural language processing (NLP) and multiagent systems has the potential to significantly enhance human-robot collaboration. A modular pipeline is proposed that enables control of multiagent systems using natural language input, designed to be compatible with energy-efficient edge hardware and suitable for deployment on devices constrained by size, weight, and power (SWaP). This pipeline consists of three key components: 1) a LLM to extract relevant information from human input, 2) a task controller to assign target states to individual agents and 3) Cognitive Map Learners (CML) that enable agents to navigate towards their target states by learning representations of node states and edge actions in an arbitrary bidirectional graph. The pipeline's effectiveness is demonstrated in a simulated environment, where a swarm of ground robots successfully executed simple natural language commands, such as “form a circle around agent 1” or “form a square in the center.” While the current implementation focuses on shape-object-preposition relationships, this work lays the foundation for future research on more complex natural language inputs and heterogeneous machine learning systems.