Kafka Producer Acknowledgements: A Complete Guide to Message Reliability

Acknowledgements are a critical feature of Apache Kafka’s Producer API, allowing you to control the reliability of message delivery. Properly configured acknowledgements not only ensures data integrity but also optimizes performance based on your application’s requirements. In this blog, we will dive deep into Kafka Producer Acknowledgements, their configuration, and best practices for leveraging them effectively.

What are Kafka Producer Acknowledgements?

Acknowledgements, also knows as “acks,” represent the producer’s mechanism for confirming whether a message has been successfully received and written to the target Kafka topic. They are controlled by the acks configuration parameter, which determines the level of guarantee the producer requires from the Kafka broker before considering a message as “successfully sent”. Note that the producer acknowledgments are fully managed through configuration, requiring no modifications to the application code.

Why are Acknowledgements Important?

Acknowledgements play a vital role in ensuring message reliability and consistency in distributed systems. They help you achieve:

1. Data Durability: Prevent message loss in the event of broker failures.
2. Fault Tolerance: Handle transient issues like network glitches or partition leader changes.
3. Performance Tuning: Strike a balance between throughput and reliability based on application needs.

Acknowledgement Configurations

Kafka provides 3 levels of acknowledgements through the acks parameter:

1. acks=0: Fire-and-Forget

• Behavior: The producer does not wait for any acknowledgment from the broker. Messages are considered sent once they leave the producer.
• Use Case: High-throughput applications where occasional message loss is acceptable (e.g., real-time metrics).
• Pros:
  • Minimal latency.
  • High throughput.
• Cons:
  • No guarantee of message delivery.
  • Increased risk of data loss.

2. acks=1: Leader Acknowledgement Only

• Behavior: The producer waits for an acknowledgment from the partition leader before considering the message successfully sent.
• Use Case: Applications requiring moderate reliability with better performance (e.g., logging systems).
• Pros:
  • Balances reliability and performance.
  • Lower latency than acks=all.
• Cons:
  • Potential data loss if the leader crashes before replicas sync.

3. acks=all or acks=-1: Acknowledgements from All In-Sync Replicas

This is also the default value of the acks config parameter.

• Behavior: The producer waits for acknowledgments from all in-sync replicas (ISRs) before considering the message successfully sent.
• Use Case: Applications requiring maximum reliability and durability (e.g., financial transactions).
• Pros:
  • Guarantees message durability.
  • Ensures data replication across ISRs.
• Cons:
  • Higher latency.
  • Reduced throughput.

How to Configure Acknowledgements

You can set the acks parameter in the producer configuration as follows:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

// Configure acknowledgements
props.put("acks", "1");  // leader acknowledgement

// Create producer
KafkaProducer<String, String> producer = new KafkaProducer<>(props);

// Send messages

For a complete example of a Kafka Producer implemented in Java, check out my other blog here – How to Build Your First Kafka Producer: A Step-by-Step Tutorial

Best Practices for Acknowledgements

1. Align Acknowledgements with Business Requirements:
   • Use acks=0 for non-critical data where speed matters.
   • Use acks=1 for moderate reliability needs.
   • Use acks=all for critical applications requiring maximum durability, such as financial transactions.
2. Monitor ISR Size:
   • Ensure a healthy number of in-sync replicas to maintain reliability for acks=all.
3. Tune retries and delivery.timeout.ms:
   • Combine acknowledgements with retries to handle transient failures effectively. To gain a deeper understanding of producer retries, check out my other blog here: Kafka Producer Retries: Everything You Need To Know For Reliable Streaming
4. Test Under Load:
   • Benchmark your producer’s performance with different acks settings to identify the optimal configuration.

A Note on the Retries and Acknowledgements

Note that if you want your producer to retry sending failed messages, this will only work if acks is set to 1 or all. With acks=0, no acknowledgment is required, and therefore, the producer does not know if a message has failed and will not attempt a retry at all.

To gain a deeper understanding of producer retries, check out my other blog here: Kafka Producer Retries: Everything You Need To Know For Reliable Streaming

Conclusion

By understanding and configuring the acks parameter effectively, you can strike the right balance between performance and durability for your application. Whether you prioritize speed or reliability, Kafka’s flexible acknowledgment settings empower you to build robust and efficient data pipelines. Happy streaming!