Dynamic Regulation of the Serotonin–Dopamine Interaction Within a Meta-reinforcement Learning Framework Encompassing the Prefrontal Cortex and Basal Ganglia

Action inhibition is essential for cognitive control, enabling individuals to prioritize relevant information over internal urges in response to changing demands. While current artificial agents excel in repetitive tasks, real-world scenarios often require the handling of unexpected constraints, such as the suppression of unwanted actions. Meta-learning, acting as an outer loop that regulates the reinforcement learning scheme in the inner loop of learning, facilitates adaptation in dynamic environments. Building upon our previous work,¹ where we implemented a brain-inspired meta-reinforcement learning framework for conflictual inhibition decision-making encompassing brain regions of the prefrontal cortex and the basal ganglia circuit and tested it within the NoGo and Stop-Signal Paradigms, this study introduces the following novelties. We explored the effects of changes in concentration and efficacy of the [Formula: see text]-mesocorticolimbic and [Formula: see text]-nigrostriatal pathways externally modulated by serotonin release on meta-reinforcement learning rules, thus the extent to which they affect behavioral performance during action cancellation within the Stop-Signal Paradigm. Our findings suggest that external serotoninergic modulation on these pathways asymmetrically affects behavioral performance, revealing that inhibitory behavior is primarily mediated by serotonin acting on [Formula: see text] dopamine receptors and is therefore asymmetrically influenced by changes in [Formula: see text] and [Formula: see text] efficacy. These pathways exhibit synergistic effects in response inhibition, with a predominant role for reductions in [Formula: see text] efficacy. Furthermore, we extended the meta-reinforcement learning framework by designing brain-inspired meta-learning rules that replicate the serotonin-dopamine dynamic interactions during action inhibition in a closed-loop fashion by using the Wilson-Cowan formalism² enabling a dynamic regulation of the exploration/exploitation rate [Formula: see text] meta-parameter. Our framework generates new predictive hypotheses and provides insights about the dynamic interaction between serotonin and dopamine [Formula: see text] and [Formula: see text], understanding their impact in response inhibition and, consequently, how they might be involved in impulsive behaviors. This knowledge suggests potential neural mechanisms underlying cognitive control in the brain and, at the same time, could contribute to the development of more flexible and robust artificial systems capable of adapting in real-world applications.