Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models

We propose a method to optimize language model pre-training data mixtures through efficient approximation of the cross-entropy loss corresponding to each candidate mixture via a Mixture of Data Experts (MDE).We use this approximation as additional features in a regression model, trained from observations of model loss for a small number of mixtures.Experiments with Transformer language models between 70M and 10B parameters on the SlimPajama dataset show that our method achieves significantly better performance than approaches that train regression models using only the mixture rates as input features.Combining our method with an objective that takes into account cross-entropy on end task data leads to superior performance on few-shot downstream evaluations.We also provide theoretical insights on why aggregation of data expert predictions can provide good approximations to model losses for data mixtures.