Talk Is Cheap, Show Me Your Colors: Low-Budget Consistency Auditing for Aggregated LLM Services

Aggregated large language model (LLM) services expose declared model identifiers while hiding the backend configurations that fulfill client requests. We present HuePrint, a lightweight black-box framework that audits brokered endpoint consistency via structured color elicitation (#RRGGBB and R,G,B) and continuous geometric distribution comparison without model weights or token probabilities. Across 34,528 deduplicated live cloud responses spanning 56 endpoints and 12 backends (Datasets A–E and Studies S1–S4), HuePrint lowers 30-call-per-side equal error rates over single-word baselines from 4.52–5.23% to 0.94–1.00% on the development cohort and from 6.25–8.04% to 3.49–5.00% across 20 locked endpoints, reaching 0.00% at 60–150 calls/side. Regularized RGB energy halves shallow-census error vs. exact-hex divergence (5.36% vs. 10.71%), and a hybrid color metric cuts cross-provider error from 16.67% (string MMD-U and pure energy) to 5.93%. Across three-week and cross-gateway checks, score-based color auditing keeps 0.00% ranking error where strict equality tests reject up to 42.86% of genuine pairs, and RGB geometry bounds numerical perturbations. Unified Lipschitz bounds and estimator diagnostics guide calibration: self-centering lowers clean held-out false rejections from 30% to 10%, and U-statistic energy ratios eliminate the 256-draw bootstrap. Version 2.0.0 changes: Editorial and precision refinement of the English main manuscript and supplementary material (explicit baseline and hybrid-metric attributions, clarified session-level burst-detector limitations, split dense clauses for readability, and verified ICLR 2025 proceedings pagination pp. 86369–86382 for Reference [4]). All 34,528 cloud responses, 58 SHA-256-protected data/baseline artifacts, equations, tables, and empirical numbers remain bit-for-bit unchanged from v1.0.0. This Zenodo record includes the complete preprint with supplementary material (HuePrint_Preprint.pdf, 28 pages), standalone main and supplementary PDFs, and a self-contained offline replication package (HuePrint_Replication_Package.zip) containing all 12 read-only JSONL response corpora, multilingual prompt batteries, statistical analysis scripts, unit tests, and SHA-256 integrity manifests.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24340930
Primary Topic
Adversarial Robustness in Machine Learning
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Talk Is Cheap, Show Me Your Colors: Low-Budget Consistency Auditing for Aggregated LLM Services

何喜冠
3 links
Zenodo (CERN European Organization for Nuclear Research)
Adversarial Robustness in Machine Learning
preprint

Talk Is Cheap, Show Me Your Colors: Low-Budget Consistency Auditing for Aggregated LLM Services

何喜冠
preprint en

Abstract

Aggregated large language model (LLM) services expose declared model identifiers while hiding the backend configurations that fulfill client requests. We present HuePrint, a lightweight black-box framework that audits brokered endpoint consistency via structured color elicitation (#RRGGBB and R,G,B) and continuous geometric distribution comparison without model weights or token probabilities. Across 34,528 deduplicated live cloud responses spanning 56 endpoints and 12 backends (Datasets A–E and Studies S1–S4), HuePrint lowers 30-call-per-side equal error rates over single-word baselines from 4.52–5.23% to 0.94–1.00% on the development cohort and from 6.25–8.04% to 3.49–5.00% across 20 locked endpoints, reaching 0.00% at 60–150 calls/side. Regularized RGB energy halves shallow-census error vs. exact-hex divergence (5.36% vs. 10.71%), and a hybrid color metric cuts cross-provider error from 16.67% (string MMD-U and pure energy) to 5.93%. Across three-week and cross-gateway checks, score-based color auditing keeps 0.00% ranking error where strict equality tests reject up to 42.86% of genuine pairs, and RGB geometry bounds numerical perturbations. Unified Lipschitz bounds and estimator diagnostics guide calibration: self-centering lowers clean held-out false rejections from 30% to 10%, and U-statistic energy ratios eliminate the 256-draw bootstrap. Version 2.0.0 changes: Editorial and precision refinement of the English main manuscript and supplementary material (explicit baseline and hybrid-metric attributions, clarified session-level burst-detector limitations, split dense clauses for readability, and verified ICLR 2025 proceedings pagination pp. 86369–86382 for Reference [4]). All 34,528 cloud responses, 58 SHA-256-protected data/baseline artifacts, equations, tables, and empirical numbers remain bit-for-bit unchanged from v1.0.0. This Zenodo record includes the complete preprint with supplementary material (HuePrint_Preprint.pdf, 28 pages), standalone main and supplementary PDFs, and a self-contained offline replication package (HuePrint_Replication_Package.zip) containing all 12 read-only JSONL response corpora, multilingual prompt batteries, statistical analysis scripts, unit tests, and SHA-256 integrity manifests.

Zenodo (CERN European Organization for Nuclear Research)
Yunnan University of Traditional Chinese Medicine (CN)
Adversarial Robustness in Machine Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon