Latency and cost-aware consumer group autoscaling in message broker systems

Message brokers often facilitate communication between data producers and consumers by adding variable-sized messages to ordered distributed queues. Our goal is to determine the number of consumers and consumer partition assignments needed to ensure that the data consumption rate matches the data production rate. We model this problem as a variable item size bin packing problem. As the production rate varies, new consumer–partition assignments are computed, potentially requiring the reallocation of partitions from one consumer to another. During reallocation, data in the queue are not read, leading to increased latency costs. To address this problem, we focus on the multiobjective optimization cost of minimizing the number of consumers and reducing latency. We introduce several heuristic algorithms and compare them to state-of-the-art heuristics. In our experimental setup, the proposed modified worst fit (MWF) heuristic achieves a 48% reduction, with a similar number of consumers, in comparison with the best fit decrease (BFD). In addition, MWF achieves a 99 t h percentile latency of 2.24 seconds compared with that of 364.66 with the approach by Kafka using the same number of consumers. Alternatively, to obtain a lower 99 t h percentile latency than our approach does, Kafka requires at least 60% more consumers than our method requires. • Modelling the consumer group autoscaling problem to consider variable message sizes and production rates. • Defining the Rscore metric to quantify latency between two consecutive consumer group assignments. • Exploring four variations of the bin packing approximation algorithms. • Implementing a fully functional framework for the consumer group autoscaling problem. • Evaluating the autoscaler response time when autoscaling the consumer group in a Kafka production infrastructure.