How does an offset expire for an Apache Kafka consumer group?

by

Update

Since Apache Kafka 2.1, offsets won’t be deleted as long as the consumer group is active, independent if the consumers commit offsets or not, ie, the offset.retention.minutes clocks only starts to tick when the group becomes empty (in older released, the clock started to tick directly when the commit happened).

Cf. https://cwiki.apache.org/confluence/display/KAFKA/KIP-211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets

Original Answer

Kafka, by default deletes committed offsets after a configurable period of time. See parameter offsets.retention.minutes. Ie, if a consumer group is inactive (ie, does not commit any offsets) for this amount of time, the offsets get deleted. Thus, even if the consumer is running, if it does not commit offsets for some partitions, those offsets are subject to offset.retention.minutes.

If you start a consumer, the following happens:

  1. look for a (valid) committed offset (for the consumer group)
    1. if valid offset is found, resume from there
    2. if no valid offset is found, reset offset according to auto.offset.reset parameter

Thus, if your offsets got deleted and auto.offset.reset = latest, you consumer will not poll anything until new data is added to the topic. If auto.offset.reset = earliest it should consume the whole topic.

See this JIRA for a discussion about this https://issues.apache.org/jira/browse/KAFKA-3806 and https://issues.apache.org/jira/browse/KAFKA-4682

More Related Contents:

  1. How to send final kafka-streams aggregation result of a time windowed KTable?
  2. Streamparse wordcount example
  3. Difference between session.timeout.ms and max.poll.interval.ms for Kafka >= 0.10.1
  4. Is there a way to purge the topic in Kafka?
  5. How to commit manually with Kafka Stream?
  6. Increase the number of messages read by a Kafka consumer in a single poll
  7. Kafka how to read from __consumer_offsets topic
  8. How to change the number of replicas of a Kafka topic?
  9. Delete message after consuming it in KAFKA
  10. Understanding Kafka Topics and Partitions
  11. When to use RabbitMQ over Kafka? [closed]
  12. Delete topic in Kafka 0.8.1.1
  13. Kafka Stream Suppress session-windowed-aggregation
  14. Use Confluent Hub without Confluent Platform installation
  15. How to change start offset for topic?
  16. Kafka 0.11 how to reset offsets
  17. How to install Kafka on Windows?
  18. How to create a Topic in Kafka through Java
  19. How to read records in JSON format from Kafka using Structured Streaming?
  20. Kafka SASL zookeeper authentication
  21. Java, How to get number of messages in a topic in apache kafka
  22. How to Delete a topic in apache kafka [duplicate]
  23. In Apache Kafka why can’t there be more consumer instances than partitions?
  24. Kafka access inside and outside docker [duplicate]
  25. How to get kafka consume lag in java program
  26. Handling exceptions in Kafka streams
  27. Is it possible to access message headers with Kafka Streams?
  28. How to get Kafka offsets for structured query for manual and reliable offset management?
  29. Kafka Consumer CommitFailedException
  30. Pyspark 2.4.0, read avro from kafka with read stream – Python

Leave a Comment