Energy-Efficient Right-Sizing of Kafka-like Message Brokers for IoT Workloads

IoT data pipelines rely on message brokers, such as Apache Kafka and Redpanda, for continuous telemetry ingestion. When it comes to capacity planning of these systems, the absence of clear sizing guidance often leads to conservative over-provisioning and unnecessary energy use. We present a calibration-based methodology for energy-efficient right-sizing of Kafka-compatible clusters for IoT ingest. Using a small set of initial experiments on 3--4 nodes, we fit a performance model that predicts maximum sustainable throughput and per-node power, enabling operators to choose the smallest cluster that satisfies a target ingest rate with headroom while minimizing energy consumption. We substantiate the approach with an experimental study of Kafka and Redpanda across three hardware generations (HDD, SATA SSD, NVMe), varying partition counts, node counts, and resource limits. We find that storage technology is the primary determinant of throughput, horizontal scaling is near-linear, and vertical CPU scaling yields diminishing returns; the two brokers exhibit distinct energy proportionality properties. On previously unseen hardware, the model predicts throughput and power with median errors below 10% and 7%, respectively. Our results provide a practical, reproducible capacity-planning workflow that maps IoT workload requirements (message size and rate) to concrete, energy-aware deployment decisions.