Benchmarking Generative AI Protein Models Reveals Differences Between Structural and Sequence-based Approaches

Recent advances in artificial intelligence have led to the development of generative models for de novo protein design. We compared 13 state-of-the-art generative protein models, assessing their ability to produce feasible, diverse, and novel protein monomers. Structural diffusion models generally create designs with higher confidence in predicted structures and more biologically plausible energy distributions, but exhibit limited diversity and strong sequence biases. Conversely, protein language models generate more diverse and novel designs but with lower structural confidence. We also evaluated these models' ability to generate unique proteins, conditionally based on the Tobacco Etch Virus (TEV) protease. Generative models were successful in producing functional enzymes, albeit with diminished activity compared to the wildtype TEV. Our systematic benchmarking provides a foundation for evaluating and selecting generative protein models, while highlighting the complementary strengths of different generative paradigms. This framework will facilitate an informed application of these tools for bio-medical engineering and design.