Recursive Estimation of Statistical Metrics over Large Datasets or Real-Time Signals

Today, digital instrumentation and the generation and analysis of data dominate the global economic landscape. Although there is no doubt that processing such data is crucial, certain statistical metrics still rely on conventional techniques that are not computationally efficient. This study introduces a median "filter" utilizing adaptive histograms (AH) that functions recursively without the requirement of storing and sorting all data. The introduced method can be utilized to estimate statistical measures for very large and expanding datasets. Alternatively, a windowed AH is also proposed to achieve online signal processing without arbitrarily discarding past values and addressing asymmetric (skewed) noise. It was subsequently shown that the method is feasible for implementation on a digital device, with its execution time assessed and its performance compared against a widely-used filtering alternative. The results showed that AHs are time-effective, accurate, and robust against uneven sampling and loss of data packets. This highlights the enabled possibility of computing statistical metrics online or over large datasets.