Behavioral Observability of Large Language Model Systems

Large language model evaluation typically reports properties of a fully configured endpoint, although observed behavior can depend on base weights, post-training, privileged instructions, tools and retrieval, conversational context, decoding, and the measurement procedure itself. This paper develops a framework for behavioral observability that treats an evaluation battery as an observation channel over system configurations. Two configurations are behaviorally equivalent relative to a battery when the battery induces indistinguishable output laws; a target property is identifiable only when it is constant on those equivalence classes. This formulation separates instrument blindness from latent capability, supports controlled factorial decomposition of deployment layers, and connects psychometric validity to observer-relative identifiability. We specify reliability and validity requirements, a statistically defensible dimensionality workflow, complementary-battery experiments, and internal-to-behavior causal bridge tests. Reanalysis of an existing 30-probe GPT-2 pilot shows why this distinction matters: five of 25 hand-coded features were constant, a fixed singular-value cutoff reported 19 effective dimensions, while permutation parallel analysis retained three components and bootstrap resampling placed the retained rank primarily between two and four. A two-model bridge pilot also showed opposite movement in internal and behavioral spectral spread. These results are diagnostic rather than confirmatory; they motivate a measurement program in which constructs, rank, layer effects, and claims of latent behavior are all validated against explicit alternatives.