UNIQUE: A Framework\nfor Uncertainty Quantification\nBenchmarking

Machine learning (ML) models have become key in decision-making\nfor many disciplines, including drug discovery and medicinal chemistry.\nML models are generally evaluated prior to their usage in high-stakes\ndecisions, such as compound synthesis or experimental testing. However,\nno ML model is robust or predictive in all real-world scenarios. Therefore,\nuncertainty quantification (UQ) in ML predictions has gained importance\nin recent years. Many investigations have focused on developing methodologies\nthat provide accurate uncertainty estimates for ML-based predictions.\nUnfortunately, there is no UQ strategy that consistently provides\nrobust estimates about model’s applicability on new samples.\nDepending on the dataset, prediction task, and algorithm, accurate\nuncertainty estimations might be unfeasible to obtain. Moreover, the\noptimum UQ metric also varies across applications, and previous investigations\nhave shown a lack of consistency across benchmarks. Herein, the UNIQUE\n(UNcertaInty QUantification bEnchmarking) framework is introduced\nto facilitate a comparison of UQ strategies in ML-based predictions.\nThis Python library unifies the benchmarking of multiple UQ metrics,\nincluding the calculation of nonstandard UQ metrics (combining information\nfrom the dataset and model), and provides a comprehensive evaluation.\nIn this framework, UQ metrics are evaluated for different application\nscenarios, e.g., eliminating the predictions with the lowest confidence\nor obtaining a reliable uncertainty estimate for an acquisition function.\nTaken together, this library will help to standardize UQ investigations\nand evaluate new methodologies.