A MCDM-Based Framework for Quantifying Reproducibility Readiness in Machine Learning Research

Machine Learning (ML) is increasingly used across scientific domains, raising concerns about the reproducibility of published results. Reproducibility is a fundamental principle of scientific research, yet many ML research works remain difficult to reproduce due to missing artifacts and insufficient reporting. This study addresses the lack of practical quantitative methods for assessing reproducibility in ML research by proposing a paper-level evaluation framework based on Multi-Criteria Decision Making (MCDM). Through a synthesis of theoretical and data-driven analyses, we identified seven key reproducibility criteria: data and code availability, the inclusion of a README and trained models, hyperparameter and training descriptions, and paper readability. These criteria are then aggregated into a unified quantitative score for ‘R1’ level reproducibility readiness using the Weighted Sum Model (WSM) and the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS). In the baseline evaluation, equal criterion weights were employed, while the Analytic Hierarchy Process (AHP) was demonstrated as a practical approach for deriving context-dependent weights tailored to diverse stakeholder priorities. The framework was applied to an annotated set of 139 randomly sampled ML research papers and benchmarked against an existing reproducibility label, achieving an accuracy of 0.64, a precision of 0.40, and a recall of 0.70. These results demonstrate that MCDM provides a feasible, interpretable, and flexible foundation for quantifying reproducibility readiness, allowing the assessment to be adapted to different decision-making contexts through customized criterion weighting.