Coordinated Online Reinforcement Learning for Self-Adaptive Systems Using Factored Q-Learning

In shared environments, the interactions between self-adaptive systems may be uncertain at design-time. This uncertainty may lead to ineffective adaptations at runtime. To address this challenge, emerging techniques use Online Reinforcement Learning (Online RL), where multiple systems learn the suitability of adaptations from numeric rewards for past adaptations. However, these existing approaches have limitations: (1) they do not coordinate adaptation selection, leading to slow learning and (2) they do not use fine-grained shared goal specification, leading to difficulties in specifying rewards. To address these limitations, we propose CoSARL, a coordinated Online RL technique for multiple self-adaptive systems. CoSARL expresses learning as interconnected local and shared learning sub-tasks. Local sub-tasks target individual system adaptation goals (e.g., minimizing energy consumption), while shared sub-tasks address goals shared by multiple systems (e.g., avoiding conflicts). Using a variant of Q-Learning, CoSARL maps the sub-tasks to individual Q-functions. Using an action coordination algorithm, the systems derive optimal adaptations, while only sharing information concerning the shared sub-tasks. We experimentally evaluate CoSARL in the DingNet exemplar and show its feasibility.