Meta-Learning Neural Mechanisms rather than Bayesian Priors

Children acquire language despite being exposed to several orders of magnitude less data than large language models require.Metalearning has been proposed as a way to integrate human-like learning biases into neuralnetwork architectures, combining both the structured generalizations of symbolic models with the scalability of neural-network models.But what does meta-learning exactly imbue the model with?We investigate the meta-learning of formal languages and find that, contrary to previous claims, meta-trained models are not learning simplicity-based priors when metatrained on datasets organised around simplicity.Rather, we find evidence that meta-training imprints neural mechanisms (such as counters) into the model, which function like cognitive primitives for the network on downstream tasks.Most surprisingly, we find that meta-training on a single formal language can provide as much improvement to a model as meta-training on 5000 different formal languages, provided that the formal language incentivizes the learning of useful neural mechanisms.Taken together, our findings provide practical implications for efficient meta-learning paradigms and new theoretical insights into linking symbolic theories and neural mechanisms.