Exact Reproduction, an Untested Promotion Path, and a Judge That Never Answered: A Pre-registered Study of MLOps Evidence on a Local Stack

Two pieces of evidence decide whether a machine learning team can be trusted near consequential decisions: a tracked, registered model line whose published metrics survive a from-scratch rerun, and a standardized evaluation whose failure cases are published rather than trimmed. This study pre-registered both tests on a local stack, in a protocol committed as the repository's root commit before any measured run existed. Half 1 re-implemented a published credit scorecard protocol (credit-risk-lab, pinned at commit fee580d0) under MLflow tracking, trained four candidates on the frozen split (30,000 accounts; 21,000 train, 9,000 test), and applied a frozen promotion gate. The champion rerun reproduced the published metrics exactly: AUC 0.7810 against a published 0.7810 (frozen tolerance 0.005, measured delta 0.0000), KS 0.4196 against 0.4196, Brier 0.1355 against 0.1355, and the logistic regression baseline also reproduced exactly (AUC 0.7518). H1 is CONFIRMED. The gate blocked both candidates it was given: the tuned challenger (AUC 0.7756) missed the frozen bar (champion + 0.003 = 0.7840), and the deliberately degraded negative control (labels permuted, AUC 0.4828) failed both frozen criteria. No candidate qualified, so the gate's promote direction was never exercised and H2 is INCONCLUSIVE under the frozen verdict logic; the challenger was not re-tuned to manufacture a pass. Half 2 built a complete Ragas evaluation of an extractive retrieval pipeline (rag-foundation, pinned at commit 007e3323) over a frozen 40-question set (28 in-corpus, 12 adversarial), and local generation already showed the targeted failure mode: the pipeline returned non-empty answers to 10 of the 12 adversarial questions whose correct behaviour is to decline. Scoring never ran: the single judge and embedding route the protocol permits returned HTTP 429 on every generateContent attempt, 21 of 21 logged calls after three successful wiring smoke calls, across the registered backoff ladder and spaced probes, and the run was stopped at 0 of 160 metric jobs. H3 and H4 are therefore NOT RUN, and no Ragas score is claimed anywhere in this paper. For this preprint, Half 1 was additionally re-executed from scratch in a clean MLflow store: every candidate's metrics and calibration table reproduced the committed record exactly, and the gate reproduced both block decisions with identical reasons.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24328468
Primary Topic
Scientific Computing and Data Management
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Exact Reproduction, an Untested Promotion Path, and a Judge That Never Answered: A Pre-registered Study of MLOps Evidence on a Local Stack

Jesun Ahmad Ushno
4 links
Zenodo (CERN European Organization for Nuclear Research)
Scientific Computing and Data Management
preprint

Exact Reproduction, an Untested Promotion Path, and a Judge That Never Answered: A Pre-registered Study of MLOps Evidence on a Local Stack

Jesun Ahmad Ushno
preprint en

Abstract

Two pieces of evidence decide whether a machine learning team can be trusted near consequential decisions: a tracked, registered model line whose published metrics survive a from-scratch rerun, and a standardized evaluation whose failure cases are published rather than trimmed. This study pre-registered both tests on a local stack, in a protocol committed as the repository's root commit before any measured run existed. Half 1 re-implemented a published credit scorecard protocol (credit-risk-lab, pinned at commit fee580d0) under MLflow tracking, trained four candidates on the frozen split (30,000 accounts; 21,000 train, 9,000 test), and applied a frozen promotion gate. The champion rerun reproduced the published metrics exactly: AUC 0.7810 against a published 0.7810 (frozen tolerance 0.005, measured delta 0.0000), KS 0.4196 against 0.4196, Brier 0.1355 against 0.1355, and the logistic regression baseline also reproduced exactly (AUC 0.7518). H1 is CONFIRMED. The gate blocked both candidates it was given: the tuned challenger (AUC 0.7756) missed the frozen bar (champion + 0.003 = 0.7840), and the deliberately degraded negative control (labels permuted, AUC 0.4828) failed both frozen criteria. No candidate qualified, so the gate's promote direction was never exercised and H2 is INCONCLUSIVE under the frozen verdict logic; the challenger was not re-tuned to manufacture a pass. Half 2 built a complete Ragas evaluation of an extractive retrieval pipeline (rag-foundation, pinned at commit 007e3323) over a frozen 40-question set (28 in-corpus, 12 adversarial), and local generation already showed the targeted failure mode: the pipeline returned non-empty answers to 10 of the 12 adversarial questions whose correct behaviour is to decline. Scoring never ran: the single judge and embedding route the protocol permits returned HTTP 429 on every generateContent attempt, 21 of 21 logged calls after three successful wiring smoke calls, across the registered backoff ladder and spaced probes, and the run was stopped at 0 of 160 metric jobs. H3 and H4 are therefore NOT RUN, and no Ragas score is claimed anywhere in this paper. For this preprint, Half 1 was additionally re-executed from scratch in a clean MLflow store: every candidate's metrics and calibration table reproduced the committed record exactly, and the gate reproduced both block decisions with identical reasons.

Zenodo (CERN European Organization for Nuclear Research)
Scientific Computing and Data Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon