rAck: proposta de configuração para garantia de entrega de mensagens no Apache Kafka

Abstract

Nowadays, one of the ways to achieve scalability and flexibility during implementation of applications is by adopting the microsservices paradigm. Through microservices, applications can be fragmented into isolated and independent processes that communicate with each other. However, a concern present in the microservice paradigm specifically deals with the communication between microservices of a same application. To solve this concern, the Apache Kafka messaging concept can be applied to exchange messages between each service. Apache Kafka is a messaging and data streaming platform that follows a producerconsumer model. Kafka’s architecture relies on the Positive Recognition (ack) mechanism, which aims to guarantee the delivery of messages. Although there are three configuration levels for ack, they present reliability or performance restrictions during the transmission of messages in unstable networks, forcing users to prioritize one of these requirements. This work aims to present Reliable Ack (rAck), a configuration for message transmission based on monitoring, identification, and recovery of messages in case of loss. An experimental phase was conducted to compare the rAck configuration with the standard ack levels in terms of reliability and performance. The experiments were structured to carry out message transmissions with the introduction of network failures that resulted in packet loss and delay in delivery. In the packet loss scenarios, the rAck configuration showed better performance when compared to Kafka’s default levels. In scenarios with delays in the delivery of packages, the introduced failures did not impact the default levels, maintaining better performances than rAck configuration. Regarding reliability, in both scenarios, the rAck configuration was able to recover all lost messages. In contrast, the default levels were subject to message loss or duplication.