Local Mechanisms of Sequence Learning in Biologically Plausible Neural Circuit Models

Abstract

The most astonishing feature of animals is their ability to interact with the external world continuously. During this continuous interaction, areas related to motor and memory functions need to learn and generate sequences that ultimately control the actions. In turn, sensory areas may need to extract from sensory feedback helpful information for dynamically adapting to the environment. These processes are typically modeled under the frameworks of unsupervised learning and reinforcement learning (RL). Yet, the implementation details of these sequence learning mechanisms in biological circuits remain poorly understood. We explored this question in both generic networks consisting of populations of excitatory and inhibitory neurons (E-I network) and biologically plausible models of zebra finch song learning. In an E-I network, we systematically examined the ability of different Hebbian learning rules to store and retrieve sequences. Using both analytical calculation and iterative optimization, we discovered that Hebbian learning of sequences within the synaptic connections between excitatory and inhibitory neurons supports higher sequence memory capacity and better robustness than learning within the recurrent excitatory synapses. Then, through the lens of zebra finch song learning, we further studied how sequence generation can be learned via self-guided reinforcement learning, investigated in two parts: the critic system that bootstraps the performance error code, and the actor system that adaptive alters the produced behaviors. For the critic, we developed circuit models suggesting that the error code arises from learning to predicatively cancel an auditory copy of the tutor song via anti-Hebbian learning. This error code resembles experimental recordings and is capable of guiding reinforcement learning of song syllable generation. For the actor, we propose that the song-specific basal ganglia serves as a reward gradient server to support the RL of sequence learning. We developed a model matching the anatomy of the song circuitry and the functional roles implied by the available evidence. We demonstrated that this model has achieved equal or better performance than the classical models of songbird RL in test environments and possesses theoretical advantages. Finally, we used a data-constrained network of HVC to illustrate how dopamine can also act as a meta-factor affecting sequence learning and generation. Together, using both generic models and models constrained to the song learning circuitry in zebra finches, this work suggests detailed theories of sequence learning in biological circuits.