Public Jailbreak Claims Are Not Reproducible Evidence: A Deployment-Realistic, Provenance-Stratified Audit of Open-Web Techniques Against a Patch-Immune Anchor

A defender choosing which jailbreaks to test against their own deployment faces a sourcing problem before a security problem. Most techniques they meet live in the grey literature, a GitHub gist, a Reddit thread, a screenshot on X, and arrive with a success rate attached, “works 100% of the time,” “95% bypass,” almost never with the prompt, model, or date that would let anyone check it. We ask whether those claimed rates are usable evidence. We take 459 techniques, harvested continuously from six open-web source types, and replay each as-published against current models in a realistic deployment configuration (model × system prompt × tools), aimed at its own native objective, judged by a consummation-gated detector built to under-count. They mostly do not survive. Across 301 techniques and 10,244 trials (of 369 carrying at least one measured trial), reproduction collapses from 40.2% (works against at least one of five panel models) to 9.0% against a frozen open-weight anchor that cannot have been silently patched, to 3.7% against the most robust model we test. The frozen anchor is the load-bearing number: its weights are fixed, so its collapse cannot be a quietly patched endpoint. Among the 17 techniques claiming roughly 100% success, only six reproduce at all, and the mean measured rate across all 17 is 13.5%. The claimed rate itself carries no usable signal. Across the 56 techniques with both a quantified claim and a measurement, claimed and measured rates show no usable relationship (Spearman’s ρ = −0.07, 95% CI [−0.34, 0.19]: the absence of a usable positive signal in this corpus, not a measured negative one). The one prior community-corpus study reports a positive correlation, but on an orthogonal construct (a researcher-assigned risk score, not a source’s own claimed rate), so it does not contradict our null; re-judging their corpus under our detector with their objective held fixed rules the detector out (the two judges agree on 96% of responses), leaving the construct difference (along with the corpus and model set) as what separates the results. The collapse is structured, not uniform. Community-sourced techniques reproduce worse than paper-sourced ones, a separation that widens as the target hardens (suggestive rather than settled at our paper-source sample, n=79, with overlapping intervals), and a per-objective decomposition (resolving the mix with an LLM reader) shows the collapse holds within every stratum, steepest for overtly harmful techniques, so it is not an artifact of mixing easy and hard objectives. The result is scoped to grey-literature techniques that survive canonicalization into a structured attack grammar (bespoke, image-only, and model-internal techniques are under-represented) and to the deployment configurations and model panel we test. Within that scope the contribution is a measurement one: a publicly claimed success rate is not reproducible evidence that a technique works, and the open-web corpus carries no mechanism that would make such a claim trustworthy without an independent replay.