A large-scale benchmark shows lightweight models can distinguish matched from mismatched problem–solution pairs across diverse STEM disciplines
Verifying that a proposed solution truly resolves a scientific problem is central to trustworthy reasoning and retrieval. Using SCP-116K, we build 177,836 balanced problem-solution pairs (88,918 matched, 88,918 mismatched) spanning diverse STEM disciplines, and frame verification, following TRIZ/IDM, as distinguishing matched from mismatched pairs. Comparing lexical, retrieval-style, and lightweight neural models, our best model (RoBERTa + Slim ResNet, frozen sentence embeddings scored by a residual MLP) reaches an AUC of 0.966, an F1 of 0.905, and a LogLoss of 0.238. A CPU-friendly TF-IDF + Cosine + Elastic-Net baseline trails by 1.6-1.7 AUC points yet runs roughly 250× faster in about 1.5 GB of RAM, a strong efficiency-accuracy trade-off. The probabilities act as re-ranking scores over candidate solutions; we read the high ROC-AUC as pairwise discrimination and absolute accuracy as an upper bound given the synthetic negatives.
Authors
- Denis Cavallucci (ORCID: https://orcid.org/0000-0003-1815-5601)
- Nicolas Douard (ORCID: https://orcid.org/0000-0002-9999-0125)
- Ahmed Samet (ORCID: https://orcid.org/0000-0002-1612-3465)
- George Giakos
Institutions
- Institut National des Sciences Appliquées de Strasbourg (FR)
- Manhattan University (US)
Publication Details
- Journal
- Scientific Reports
- Published
- 2026-06-30
- DOI
- https://doi.org/10.1038/s41598-026-60058-z
- Primary Topic
- Model Reduction and Neural Networks
- Type
- article
- Field-Weighted Citation Impact
- 0.00