Measuring a Multi-Agent LLM Pipeline for Scientific Contradiction Detection and Hypothesis Generation

Large language models are increasingly proposed as engines of scientific discovery, but claims about their output are rarely accompanied by measurements of how often that output is wrong. We built a four-stage pipeline that ingests physics literature, extracts atomic claims, detects contradictions between them, generates candidate resolutions, and critiques those resolutions automatically — then measured every stage, including the stages that failed. Contradiction detection works: 76% recall (95% CI 60–88%) against ground truth taken from published arXiv "Comment on X" / "Reply to X" disagreements, at 0 false positives across 120 similarity-matched negative controls. Hypothesis generation does not. In a pre-registered random sample of 60 hypotheses, 35% (95% CI 24–48%) contained an error checkable against textbook physics. An independent model rating, blind to our labels, found 33%. The error rate ranges from 0% to 67% depending on the dispute, and the characteristic failure is using a real tool outside its domain of validity. Automated critique could not be shown to work: the critic accepted 87% of everything, and its verdicts are not significantly associated with correctness (Fisher exact, p = 0.09). A targeted prompt intervention aimed at the dominant failure mode produced no significant change (McNemar p = 0.405) despite full compliance. Across every stage, the improvements that mattered came from specification and data quality rather than model capacity, and three successive versions of one automated gate were structurally incapable of failing while appearing to pass. Total API cost: $1.90. Code, database, and all labels are published.