ML/NLP in the biological domain - adapting to large, complex datasets

The Department of Energy Systems Biology Knowledgebase (KBase) explores novel machine learning and natural language processing (ML/NLP) use-cases in the biological domain. Due to the scale, complexity, and non-uniformity of the data within the KBase platform, existing ML/NLP pipelines must be adjusted to meet these challenges. This work will detail several of these concerns and solutions in the context of biological research in DOE-centric scientific focus areas that are likely shared by many domains outside of biology. While mining training data from numerous and diverse sources is a common first step in ML projects, mixed data sources can complicate the interpretation of the results. In particular, KBase has developed a model for classifying annotated metagenomes with the environment from which they were extracted using gradient-boosted decision trees (Catboost1). This model was trained on the MGnify dataset, a complex data source comprising many different sources of annotations, including taxonomic labels, protein domain and full-length functional annotations using Gene Ontology2,3 and InterPro4 annotations. Deriving a meaningful interpretation of the data relies on downstream analysis after model training and evaluation, such as permutation and feature importance analysis. Results from NLP classification tasks, including those of biological relevance, are often improved by augmenting input with domain-specific context (RAG5, etc.). This requirement can preclude the use of models with small context windows. The size of the model also directs a lab’s ability to use it, as very large models may require prohibitively expensive pre-training or fine-tuning. In our work, we explore the results of using different models with varying context window sizes and their ability to improve classification metrics for NLP models trained for genetic tool development and biomanufacturing tasks. Sampling bias may occur as a result of slight variations in experimental protocols and data sources. These issues impact a model’s ability to generalize to data on which it was not trained. In biological applications, like phenotype classification, these variations can impact the predictive performance for out-of-clade predictions by machine learning classifiers6. Addressing these effects calls for innovative sampling techniques that extend beyond basic random and stratified splitting. We are developing a sampling technique based on the similarity of the most important and predictive features that use phenotypic similarity as a proxy. While issues in developing a useful ML pipeline or NLP model are shared across many domains, the exact means of mitigating is field-dependent. KBase’s research, focusing on biological applications, is also subject to these issues, often requiring insight into the biological domain to guide model result improvement.