CFC Cross-Model Benchmark v1: Claude and Gemini Evaluation (600 Primary Runs)

I’m sharing the frozen CFC Cross-Model Benchmark v1 evaluation covering 600 primary runs: 300 Claude runs and 300 Gemini runs across the same 100 decision-closure variants. Both model environments achieved the same strict semantic PASS rate: 296/300 (98.67%). However, the observed semantic anomalies occurred on different variants and through different mechanisms. Claude’s failures included reasoning attribution, stale-state carryover, instruction-recognition failure, and false closure from duplicate priority coverage. Gemini’s failures were dominated by state/layer binding errors. The anomaly variants did not overlap in these frozen runs. I treat that as a descriptive observation, not as evidence of systematic model complementarity. The package also preserves the provenance limitation for Gemini CM-V002 and CM-V003. Excluding all six associated runs gives 290/294 semantic PASS (98.64%), so the aggregate conclusion is materially unchanged. The narrow conclusion is that a near-99% semantic PASS rate on this benchmark does not eliminate rare state-transition, state-binding, instruction-following, and decision-closure failures. This does not demonstrate that CFC prevents these errors. That requires a separate controlled comparison using the same frozen benchmark with and without the CFC/controller layer. I’m sharing the package as a baseline reliability record and as groundwork for that next intervention study. Where preserved and provenance-verified, the full benchmark questions/prompts and corresponding Claude and Gemini responses may be made available on request for independent inspection, scoring review, or methodological audit. Any such release would preserve the original frozen materials and would clearly identify any items for which source provenance could not be independently verified.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-08-26
DOI
https://doi.org/10.5281/zenodo.22117671
Primary Topic
Scientific Computing and Data Management
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

CFC Cross-Model Benchmark v1: Claude and Gemini Evaluation (600 Primary Runs)

Krzysztof Jan Sliwka
Zenodo (CERN European Organization for Nuclear Research)
Scientific Computing and Data Management
article

CFC Cross-Model Benchmark v1: Claude and Gemini Evaluation (600 Primary Runs)

Krzysztof Jan Sliwka
article en

Abstract

I’m sharing the frozen CFC Cross-Model Benchmark v1 evaluation covering 600 primary runs: 300 Claude runs and 300 Gemini runs across the same 100 decision-closure variants. Both model environments achieved the same strict semantic PASS rate: 296/300 (98.67%). However, the observed semantic anomalies occurred on different variants and through different mechanisms. Claude’s failures included reasoning attribution, stale-state carryover, instruction-recognition failure, and false closure from duplicate priority coverage. Gemini’s failures were dominated by state/layer binding errors. The anomaly variants did not overlap in these frozen runs. I treat that as a descriptive observation, not as evidence of systematic model complementarity. The package also preserves the provenance limitation for Gemini CM-V002 and CM-V003. Excluding all six associated runs gives 290/294 semantic PASS (98.64%), so the aggregate conclusion is materially unchanged. The narrow conclusion is that a near-99% semantic PASS rate on this benchmark does not eliminate rare state-transition, state-binding, instruction-following, and decision-closure failures. This does not demonstrate that CFC prevents these errors. That requires a separate controlled comparison using the same frozen benchmark with and without the CFC/controller layer. I’m sharing the package as a baseline reliability record and as groundwork for that next intervention study. Where preserved and provenance-verified, the full benchmark questions/prompts and corresponding Claude and Gemini responses may be made available on request for independent inspection, scoring review, or methodological audit. Any such release would preserve the original frozen materials and would clearly identify any items for which source provenance could not be independently verified.

Zenodo (CERN European Organization for Nuclear Research)
Peace, Justice and strong institutions
Openalex Percentile: Top 4%
Scientific Computing and Data Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

CFC Cross-Model Benchmark v1: Claude and Gemini Evaluation (600 Primary Runs) — Krzysztof Jan Sliwka · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS