Aggregable Generalized Deduplication

Considering the edge-cloud environment and uploading large amounts of data from IoT devices through the edge node towards the cloud, this paper investigates an aggregation method of data streams compressed by Generalized Deduplication (GD) for the edge node. The simplest way is to decode GDcompressed data streams, aggregate raw streams and re-encode it into a single GD-compression stream. However, this involves a large computational complexity for aggregation due to the decoding and re-encoding of linear codes underlain the GD. From this observation, this paper presented a novel aggregation method, called Aggregable GD (AGD). The AGD is designed to aggregate multiple GD streams into a single AGD stream, and removes duplicated information among GD streams without decoding and re-encoding operations of GD. This paper also shows that AGD involves smaller computational complexity for data aggregation than the ordinary simple scheme. Furthermore, by the preliminary computer simulation using the Hamming code as the underlying linear code of GD, we demonstrate that AGD performs comparable to the ordinary method from the viewpoints of the compression rate.