Extracting, Ablating, and Transferring Language Model Capabilities

This thesis explores the understanding, extraction, ablation, and transfer of language model (LM) capabilities. Understanding how training data influences model capabilities, extracting the specific capabilities present in a given model, and ablating unwanted behaviors is important for the quality of LMs. Transfer of capabilities between models serves this purpose as well in addition to effectively reusing the spent compute used to train constituent models and freeing up the development workflows of such models. The first part of this thesis deals with how capabilities get imbued into the model parameters via training. I explore the relationship between training data and model parameters via theoretical developments on the optimization of shallow ReLU networks. Namely, I characterize the loss landscape using a geometric and combinatorial structure known as a zonotope constructed from the training data. This has relevance for the capabilities stored in transformer feedforward (FFW) layers, which are shallow neural networks. The second part of this thesis demonstrates how to extract fine-grained capabilities and selectively ablate them. Non-negative per-example Fisher factorization (NPEFF) finds patterns in the relationship between model parameters and predictions that correspond to interpretable behaviors. Notably, it provides a principled way to selectively disrupt behaviors by perturbing their corresponding parameters. The last part of this thesis deals with transfer of capabilities between models via model merging. I introduce Fisher-weighted model averaging, which phrases model merging as likelihood maximization using the Laplace approximation to model posteriors. This provides an alternative that outperforms simple weight averaging. Through developing the understanding, extraction, ablation, and transfer of LM capabilities, this thesis provides tools to better understand and modify LMs in addition to increasing the flexibility and decreasing the resource requirements of their development.