Holding an open validation engine to outside references: five ECB statistics, a credit-portfolio engine, an IRRBB engine and twelve planted defects
Validators of bank risk models need tests that supervisors recognise, reference calculations to compare with, and evidence that both are correct. We describe an open engine with 134 registered tests, five of the 17 ECB statistics for validating internal credit models, a copula and CreditRisk+ portfolio engine, an interest rate risk engine for the banking book, and a protocol that holds each component to references outside it; two references share the engine's arithmetic. The five ECB statistics agree with an independent transcription of the instructions within 4.4 x 10^-11 on 500 random inputs each; the Jeffreys test differs from the exact binomial test in 1 of 70 grades at the 5 % level. The portfolio engine matches four analytic anchors, an independent simulator (largest |z| below one in 10 runs of 200,000 scenarios), an independent recursion and its committed one-million-scenario rung (relative 2.9 x 10^-9); the interest rate engine matches QuantLib (relative 2.4 x 10^-14, 72 leg and shock pairs). Not every hypothesis held: 11 of the 17 ECB statistics are not implemented, and the maximum-likelihood interval of the asset correlation covers below nominal in 16 of 24 simulator cells, outside the binomial band in 1. Planting 12 defects shows that the five verification layers are not interchangeable: 10 of 10 real defects are found, 8 by one layer only. The hypotheses were not pre-registered. That no open toolkit offers these tests is checked only against 34 package descriptions and 106 installed distributions. Repository and reproducible pipeline (MIT): https://github.com/fsantibanezleal/CAOS_Contraste .
Authors
- Felipe A. Santibáñez-Leal (ORCID: https://orcid.org/0000-0002-0150-3246)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-08
- DOI
- https://doi.org/10.5281/zenodo.24053177
- Primary Topic
- Credit Risk and Financial Regulations
- Type
- preprint