TooT-SS: Transfer Learning using ProtBERT-BFD Language Model for Predicting Specific Substrates of Transport Proteins

Transmembrane transport proteins are essential in cell life for the passage of substrates across cell membranes. Metabolic network reconstruction requires transport reactions that describe the specific substrate transported as well as the metabolic reactions of enzyme catalysis. We utilize a protein language model called ProtBERT (Protein Bidirectional Encoder Representations from Transformers) and transfer learning with a one-layer Feed-Forward Neural Network (FFNN) to predict 96 specific substrates. We automatically construct a dataset UniProt-SPEC-100 using the ChEBI and GO ontologies with 4,455 sequences from 96 specific substrates. This dataset is extremely imbalanced with a ratio of 1:408 between the smallest class and the largest. Our model TooT-SS predicts 83 classes out of 96 with an F1-score of 0.92 and Matthews Correlation Coefficient (MCC) of 0.91 on a hold-out test set. The results of 3-fold cross-validation experiments, particularly, on small classes show the potential of transfer learning from the ProtBERT language model for handling imbalanced datasets.