Assessing the structural novelty of AI-generated protein binders: Applying the Conservatively Weighted Mismatch metric to the BindCraft dataset

This article addresses the problem of estimating the number of truly novel structures (peptides, enzymes, proteins) obtained using AI models. Modern generative models (LigandForge, RFDiffusion, BindCraft, and others) generate peptides at record speeds, but their comparative analysis is hampered by the lack of a unified metric or system of interconnected metrics that evaluates the ability to generate new structural motifs rather than reproduce the training dataset. This article proposes the "Conservatively Weighted Dissimilarity" (CWD) metric, which measures the degree to which a peptide is "foreign" to its family, i.e., whether it uses amino acids that are rare or unusual for the given positions. The proposed metric is model-independent and can be applied to any generative approach. A formal definition is provided, and the limitations of the model are discussed. Results from applying the metric to the BindCraft dataset are presented. The mean CWD across the sample is 0.8249, indicating that the BindCraft model generates evolutionarily novel peptides with rare amino acids at the corresponding positions. We also provide Python code for calculating the conservatively weighted CWD disparity for this paper. The code takes an archive of PBD files as input and returns the CWD value for a sample.