Decision support process to define Apache Kafka producer configuration

The subject of the article is decision support methods for determining the configuration of an Apache Kafka cluster that ensures high throughput for messages from the Kafka producer to the Kafka consumer, which is one of the key requirements for information technologies managing heterogeneous data in urban construction projects. The purpose of the work is to develop a decision support process to determine the number of partitions in an Apache Kafka topic based on sensitivity analysis in a discrete Bayesian network. The process offers recommendations regarding the performance metrics of the Apache Kafka producer and their interval values that should be considered when determining the number of partitions. To achieve the stated goal, the following tasks were solved: reviewing and analyzing existing formal methods for determining the number of partitions in an Apache Kafka topic and identifying the reasons why these methods may be less effective; defining mathematical methods and developing the structure of a Bayesian network for inclusion in the decision support process for determining the number of partitions in an Apache Kafka topic; evaluating the proposed process based on the results of verification tests. To accomplish these tasks, methods from probability and statistics, systems analysis, artificial intelligence, and information technology theories were employed. Based on the results of verification tests of the proposed process, its ability to generate recommendations for determining the number of partitions in a Kafka topic has been demonstrated. Furthermore, the proposed process identifies performance metrics of an Apache Kafka producer, the values of which should be continuously monitored to maintain the producer's performance after deploying an information technology system that includes a Kafka cluster. The configuration of a Kafka topic, determined based on the formulated recommendations, will reduce the likelihood of scenarios where these configurations are revised after deploying the information system, which could lead to the risk of breaching the data availability threshold.