Before Calling It a Non-Replication: Instrument Qualification Across LLM Families

When a behavioral effect observed in one large language model is absent on another model family, the result is often described as weak generalization or non-replication. That interpretation assumes that the task remains an interpretable measurement instrument for the new model. This methodological note examines that assumption through a preserved sequence of cross-family qualification attempts following a completed pre-specified fixed-N evidence-weighting study. The original Qwen study found that five explicitly derivative reviews of one source increased retention of an initial claim relative to an equal-count control containing five unrelated memory records (22/32 versus 0/32; paired risk difference 0.6875; Holm-adjusted exact McNemar p = 9.5367e-7). Subsequent Phi and Ministral programs introduced frozen preflight, eligibility, calibration, and redesign gates before permitting confirmatory collection. None reached a valid cross-family confirmatory estimate. A post-publication no-new-data forensic validation independently reproduced the v0.4.3 H1/H2 calculations and found no material data or statistical error, while narrowing claims about preregistration, control matching, and mechanism. The note presents the full history as a case study in instrument qualification and measurement transport across LLM families and proposes a practical reporting distinction among valid replication/non-replication, configuration ineligibility under the instrument, and invalid/inconclusive execution. No new behavioral observations were generated for this manuscript. AI Use and Contribution Disclosure This research is human-led and AI-assisted. The original research topic, initial scientific question, and decision to pursue this line of investigation were proposed by Ovidiu Boticiu. ChatGPT (OpenAI) was subsequently used for methodological discussion and critique, structuring alternatives, protocol-development assistance, coding and code review, statistical and logical checks, documentation, manuscript drafting/editing, and methodological audit support. Final methodological and experimental decisions, authorization and execution of experimental runs, interpretation of results, publication decisions, and scientific responsibility remained with the human author. Experimental observations and numerical results derive from the executed protocols and model outputs, not from ChatGPT-generated claims or simulated data. ChatGPT is not listed as an author and is not treated as an independent scientific authority or evidentiary source.