What determines Kafka consumer offset?

by

It is a bit more complex than you described.
The auto.offset.reset config kicks in ONLY if your consumer group does not have a valid offset committed somewhere (2 supported offset storages now are Kafka and Zookeeper), and it also depends on what sort of consumer you use.

If you use a high-level java consumer then imagine following scenarios:

  1. You have a consumer in a consumer group group1 that has consumed 5 messages and died. Next time you start this consumer it won’t even use that auto.offset.reset config and will continue from the place it died because it will just fetch the stored offset from the offset storage (Kafka or ZK as I mentioned).

  2. You have messages in a topic (like you described) and you start a consumer in a new consumer group group2. There is no offset stored anywhere and this time the auto.offset.reset config will decide whether to start from the beginning of the topic (earliest) or from the end of the topic (latest)

One more thing that affects what offset value will correspond to earliest and latest configs is log retention policy. Imagine you have a topic with retention configured to 1 hour. You produce 5 messages, and then an hour later you post 5 more messages. The latest offset will still remain the same as in previous example but the earliest one won’t be able to be 0 because Kafka will already remove these messages and thus the earliest available offset will be 5.

Everything mentioned above is not related to SimpleConsumer and every time you run it, it will decide where to start from using the auto.offset.reset config.

If you use Kafka version older than 0.9, you have to replace earliest, latest with smallest,largest.

More Related Contents:

  1. Kafka Consumer CommitFailedException
  2. Can a Kafka consumer(0.8.2.2) read messages in batch
  3. Connect to Kafka running in Docker
  4. How can I send large messages with Kafka (over 15MB)?
  5. How to create a Topic in Kafka through Java
  6. How Can we create a topic in Kafka from the IDE using API
  7. Writing Custom Kafka Serializer
  8. Java, How to get number of messages in a topic in apache kafka
  9. How to Delete a topic in apache kafka [duplicate]
  10. When does the Apache Kafka client throw a “Batch Expired” exception?
  11. What is a simple, effective way to debug custom Kafka connectors?
  12. Is it possible to add partitions to an existing topic in Kafka 0.8.2
  13. How to get kafka consume lag in java program
  14. Change File Split size in Hadoop
  15. Handling exceptions in Kafka streams
  16. Is it possible to access message headers with Kafka Streams?
  17. Dynamically connecting a Kafka input stream to multiple output streams
  18. How to create Custom serializer in kafka?
  19. How can I delete a topic in Apache Kafka? [duplicate]
  20. Analyze the following code, which exception does it display? (Exception handling, java)
  21. Hibernate Error: org.hibernate.NonUniqueObjectException: a different object with the same identifier value was already associated with the session
  22. What is the Cost of Calling array.length
  23. Setting the Default Font of Swing Program
  24. What is the difference between CloseableHttpClient and HttpClient in Apache HttpClient API?
  25. How do we count rows using older versions of Hibernate (~2009)?
  26. LambdaConversionException with generics: JVM bug?
  27. draw polar graph in java
  28. Java HashMap nested generics with wildcards
  29. constructor of subclass in Java
  30. Creating LinearLayout Programmatically/Dynamically with Multiple Views

Leave a Comment