Stochastic Technique on Assigning Adaptive Minimal Code Bits to Locally Frequent Symbols in Continuous Data Stream

As increasing demands for recognizing social environment and/or human activity using sensory devices and video cameras, streaming data has become one of major data types. The applications process the data stream and generate new knowledge due to recent AI technologies. During the scenario, it is important to consider how fast the data stream is transferred from the sensing equipment to the computing platform. Lossless compression is one of the solutions for the fast communication, which reduces the amount of data. This paper focuses on realtime encoding mechanism for data stream by employing lossless encoding technique. Conventional encoding mechanisms such as Huffman coding use data entropy based on frequency counters of data symbols. According to the ranking of the frequencies of data unit, that is, for lower entropy, shorter code is assigned to the corresponding symbol to compress a given dataset. However, we found that the method with frequency counters can not identify actual instant entropy because its accumulated frequency includes past entropy. We also found that an encoder can assign shorter code to the locally frequent symbols suddenly appeared in data stream, and that it results more data reduction than the conventional methods. In this paper, we propose a novel stochastic method with Markov model to respond the locally frequent symbols. The method uses the entropy calculations from the state machine of Markov model with/without a frequency counting. From experimental evaluations, we show the technique follows instant entropy changes of data stream and compresses the symbols to appropriate shorter codes. Through experimental evaluations, we show validity of the technique with describing the algorithm. The experimental evaluations show improvements up to 17% better compression ratios due to the instant response to the locally frequent symbols.