An Entropy-Based Effective Support Diagnostic for Semantic Collapse in LLM Memory and Retrieval-Augmented Generation

This technical note proposes an entropy-based diagnostic for detecting semantic collapse in LLM memory and retrieval-augmented generation pipelines. Given a partition of retrieved or stored context items into semantic classes, the method defines the effective number of distinguishable semantic states as N_eff = k exp(−D_KL(ν ‖ U_k)) = exp(H(ν)), where ν is the empirical class distribution and U_k is the uniform distribution over k classes. The measure quantifies the discrepancy between nominal class count and effective semantic diversity. The note further describes how N_eff and the growth of KL divergence under a compression parameter can be used as control signals for adaptive context compression, semantic deduplication, and diversity-aware retrieval. The record includes the technical brief and a minimal Python reference implementation.