Delete message after consuming it in KAFKA

by

In Kafka, the responsibility of what has been consumed is the responsibility of the consumer and this is also one of the main reasons why Kafka has such great horizontal scalability.

Using the high level consumer API will automatically do this for you by committing consumed offsets in Zookeeper (or a more recent configuration option is using by a special Kafka topic to keep track of consumed messages).

The simple consumer API make you deal with how and where to keep track of consumed messages yourself.

Purging of messages in Kafka is done automatically by either specifying a retention time for a topic or by defining a disk quota for it so for your case of one 5GB file, this file will be deleted after the retention period you define has passed, regardless of if it has been consumed or not.

More Related Contents:

  1. Difference between session.timeout.ms and max.poll.interval.ms for Kafka >= 0.10.1
  2. Kafka how to read from __consumer_offsets topic
  3. Understanding Kafka Topics and Partitions
  4. How to send final kafka-streams aggregation result of a time windowed KTable?
  5. Streamparse wordcount example
  6. What determines Kafka consumer offset?
  7. Is there a way to purge the topic in Kafka?
  8. How does an offset expire for an Apache Kafka consumer group?
  9. How to commit manually with Kafka Stream?
  10. Increase the number of messages read by a Kafka consumer in a single poll
  11. How to change the number of replicas of a Kafka topic?
  12. When to use RabbitMQ over Kafka? [closed]
  13. Delete topic in Kafka 0.8.1.1
  14. Kafka Stream Suppress session-windowed-aggregation
  15. Kafka Consumer CommitFailedException
  16. Can a Kafka consumer(0.8.2.2) read messages in batch
  17. Use Confluent Hub without Confluent Platform installation
  18. How to change start offset for topic?
  19. Kafka 0.11 how to reset offsets
  20. How can I send large messages with Kafka (over 15MB)?
  21. Using Kafka as a (CQRS) Eventstore. Good idea?
  22. How to write spark streaming DF to Kafka topic
  23. Writing Custom Kafka Serializer
  24. Kafka server configuration – listeners vs. advertised.listeners
  25. When does the Apache Kafka client throw a “Batch Expired” exception?
  26. What is a simple, effective way to debug custom Kafka connectors?
  27. Why I cannot connect to Kafka from outside?
  28. Dynamically connecting a Kafka input stream to multiple output streams
  29. How to integrate Oracle and Kafka
  30. removing a kafka consumer group in zookeeper

Leave a Comment