ARMA Processes With Discrete-Continuous Excitation: Compressibility Beyond Sparsity

The Rényi Information Dimension (RID) is a fundamental measure for quantifying the compressibility of random variables with singularities in their distributions, extending beyond classical notions of sparsity. At a high level, RID represents the average number of bits required to encode i.i.d. samples of a random variable with high precision. For stochastic processes, two main extensions of RID exist: the information dimension rate (IDR) and the block information dimension (BID). A more recent approach to characterizing the compressibility of stochastic processes is through ϵ-achievable compression rates, which treat a random process as the limit of finite-dimensional random vectors and leverage tools from compressed sensing. However, the interplay between BID, IDR, and ϵ-achievable compression rates remains poorly understood. Furthermore, explicit values of IDR and BID are known only for a limited class of processes, such as i.i.d. sequences (i.e., discrete-time white noise) and moving-average (MA) processes. This paper investigates the IDR and BID of discrete-time Auto-Regressive Moving-Average (ARMA) processes and their relationship with ϵ-achievable compression rates when the excitation noise follows a discrete-continuous distribution. Specifically, we show that the RID and ϵ-achievable compression rates of such ARMA processes are equal to those of their excitation noise. In other words, despite the fact that ARMA process samples are not sparse, their compressibility matches that of their sparse excitation noise. To establish this result, we demonstrate that the singular components of the sample distribution are supported on affine sets, with relative dimensions that concentrate around the BID. Leveraging a known result on typical affinely singular sources, we further prove that in this setting, the RID coincides with ϵ-achievable compression rates. The findings of this paper provide new insights into the compressibility of locally correlated data with finite- or infinite-memory, which are commonly modeled using ARMA processes.