Modelling Reinforcement Learning in the Basal Ganglia: A Neuromodulated Approach on SpiNNaker

We have presented brain-inspired reinforcement learning using a model of the Basal Ganglia (BG) implemented on the neuromorphic computer SpiNNaker. The novelty of this work is the inclusion of the the Substantia Nigra pars compacta population of BG that releases the neurochemical Dopamine (DA). Neuromodulation by DA allows dynamic synaptic weight learning by rewarding desired actions. This is unlike previous models that focused on action-selection using static weights. Furthermore, we have scaled up the model such that there are two channels, each of which are trained to select specific actions in response to input cues provided by periodic spike trains. Also, both fast and slow excitatory input pathways to the BG model from the cortex are used, unlike in previous work where only the fast synapses were modelled. To test the trained network, we simulated a robotic navigation system. The model is required to demonstrate a wall-following behaviour in a room environment of different dimensions. Thus, a ‘move forward’ action is selected for no detected obstacle; a ‘turn left’ action is selected otherwise. The environment is sensed at periodic intervals and live spike injection is used to provide cue to the model to take the required action. Our results show that the two-channel BG network trained with instrumental conditioning avoided clashing into corners successfully and retained its wall-following behaviour. Overall, our work provides a neuromorphic framework for implementing robust decision-making and action-selection systems on low-power real-time neuromorphic devices.