Proteins need extra attention: improving the predictive power of protein language models on mutational datasets with hint tokens

In this computational study, we address the challenge of predicting protein functions following mutations by fine-tuning protein language models (PLMs) using a novel tokenization strategy, hint token learning (HTL). To evaluate the effectiveness of HTL, we benchmarked this approach across four pretrained models with varying architectures and sizes on four diverse protein mutational datasets. Our results showed significant improvements in weighted F1 scores in most cases when HTL was applied. To understand how HTL enhances protein mutational predictions, we trained sparse autoencoders on embeddings derived from the fine-tuned PLMs. Analysis of the latent spaces revealed that the number of activated residues within functional protein domains increased by PLM training with HTL. These findings indicate that PLMs fine-tuned with HTL may capture more biologically relevant representations of proteins. Our study highlights the potential of HTL to advance protein function prediction and provides insights into how HTL enables PLMs to capture mutational impacts at the functional level. All data and code are available at: https://github.com/ejp-lab/EJPLab_Computational_Projects/tree/master/HintTokenLearning.