DriftShard: Drift-Aware Adaptive Sharding for Distributed Industrial IoT Time-Series Databases

Abstract Distributed databases that store Industrial Internet of Things (IIoT) telemetry are usually sharded once, by device identifier or by time, and left in that state. The queries that reach them do not stay fixed: operator drill-downs, real-time monitoring and fleet-level analytics alternate over a working day, and each favours a different placement of the data. Re-sharding can recover lost performance, but moving millions of rows is expensive, and a system that re-shards on every change may spend more time migrating than it saves. This paper presents DriftShard, a framework that monitors the mix of query classes reaching a Citus cluster, detects regime changes with a total variation distance test, predicts the latency of every candidate layout with a gradient-boosted cost model, and migrates only when the predicted saving over a planning horizon exceeds the predicted migration time. DriftShard was implemented on PostgreSQL 16 with Citus 12.1 and evaluated with real migrations on a four-node cluster, using a synthetic sensor fleet (0.71 million rows) and the TON IoT telemetry collection (3.58 million rows). Across 40 randomised regime sequences, DriftShard was 42% and 30% faster than a static device-hash layout on the two datasets, and 31% and 53% faster than drifttriggered re-sharding that ignores migration cost. It did not match the best static layout chosen in hindsight, and its benefit depended on cost-model accuracy: with measured rather than predicted costs it improved by a further 6% and 13%. The results identify when adaptive sharding pays off and where learned cost models limit it.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-13
DOI
https://doi.org/10.21203/rs.3.rs-11584427/v1
Primary Topic
Advanced Database Systems and Queries
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

DriftShard: Drift-Aware Adaptive Sharding for Distributed Industrial IoT Time-Series Databases

Nalinda Somasiri, Kavindu Karunarathne
1 links
Research Square
Advanced Database Systems and Queries
preprint

DriftShard: Drift-Aware Adaptive Sharding for Distributed Industrial IoT Time-Series Databases

Nalinda Somasiri, Kavindu Karunarathne
preprint en

Abstract

Abstract Distributed databases that store Industrial Internet of Things (IIoT) telemetry are usually sharded once, by device identifier or by time, and left in that state. The queries that reach them do not stay fixed: operator drill-downs, real-time monitoring and fleet-level analytics alternate over a working day, and each favours a different placement of the data. Re-sharding can recover lost performance, but moving millions of rows is expensive, and a system that re-shards on every change may spend more time migrating than it saves. This paper presents DriftShard, a framework that monitors the mix of query classes reaching a Citus cluster, detects regime changes with a total variation distance test, predicts the latency of every candidate layout with a gradient-boosted cost model, and migrates only when the predicted saving over a planning horizon exceeds the predicted migration time. DriftShard was implemented on PostgreSQL 16 with Citus 12.1 and evaluated with real migrations on a four-node cluster, using a synthetic sensor fleet (0.71 million rows) and the TON IoT telemetry collection (3.58 million rows). Across 40 randomised regime sequences, DriftShard was 42% and 30% faster than a static device-hash layout on the two datasets, and 31% and 53% faster than drifttriggered re-sharding that ignores migration cost. It did not match the best static layout chosen in hindsight, and its benefit depended on cost-model accuracy: with measured rather than predicted costs it improved by a further 6% and 13%. The results identify when adaptive sharding pays off and where learned cost models limit it.

Research Square
York St John University (GB)
Industry, innovation and infrastructure
Advanced Database Systems and Queries
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon