How to change start offset for topic?

by

Since kafka 0.11.0.0 you can use the script kafka-consumer-groups.sh
Example from this answer

kafka-consumer-groups.sh --bootstrap-server kafka-host:9092 --group my-group --reset-offsets --to-earliest --all-topics --execute

Other options listed in the KIP-122: Add Reset Consumer Group Offsets tooling

.----------------------.-----------------------------------------------.----------------------------------------------------------------------------------------------------------------------------------------------.
|      Scenario        |                   Arguments                   |                                                                    Example                                                                   |
:----------------------+-----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------:
| Reset to Datetime    |  --to-datetime YYYY-MM-DDTHH:mm:SS.sss±hh:mm  |  Reset to first offset since 01 January 2017, 00:00:00 hrs: --reset-offsets –group test.group --topic foo --to-datetime 2017-01-01T00:00:00Z |
:----------------------+-----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------:
| Reset by Duration    |  --by-duration  PnDTnHnMnS                    |  Reset to first offset since one week ago (from current timestamp): --reset-offsets --group test.group --topic foo --by-duration P7D         |
:----------------------+-----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------:
| Reset to Earliest    |  --to-earliest                                |  Reset to earliest offset available: --reset-offsets --group test.group --topic foo --to-earliest                                            |
:----------------------+-----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------:
| Reset to Latest      |  --to-latest                                  |  Reset to latest offset available: --reset-offsets --group test.group --topic foo --to-latest                                                |
:----------------------+-----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------:
| Reset to Offset      |  --to-offset                                  |  Reset to offset 1 in all partitions: --reset-offsets --group test.group --topic foo --to-offset 1                                           |
:----------------------+-----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------:
| Shift Offset by 'n'  |  --shift-by n                                 |  Reset to current offset plus 5 positions: --reset-offsets --group test.group –topic foo --shift-by 5                                        |
:----------------------+-----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------:
| Reset from File      |  --from-file PATH_TO_FILE                     |  Reset using a file with reset plan: --reset-offsets --group test.group --from-file reset-plan.csv                                           |
'----------------------'-----------------------------------------------'----------------------------------------------------------------------------------------------------------------------------------------------'

You can also define the partition you want to reset, example:

  • Reset offset of topic foo partition 0 to 1

    --reset-offsets --group test.group --topic foo:0 --to-offset 1

  • Reset offset of topic foo partition 0,1,2 to earliest

    --reset-offsets --group test.group --topic foo:0,1,2 --to-earliest

Reminder: don’t forget the --execute flag (see the execution options in the KIP). Without this flag the script will only print out the result of the scenario by scope, for example:

TOPIC                 PARTITION NEW-OFFSET NEW-LAG LOG-END-OFFSET CONSUMER-ID HOST CLIENT-ID
foo                   0         90         10      100            -           -    -

Credits to this answer.
Table created with ascii tables

More Related Contents:

  1. How to send final kafka-streams aggregation result of a time windowed KTable?
  2. Streamparse wordcount example
  3. Difference between session.timeout.ms and max.poll.interval.ms for Kafka >= 0.10.1
  4. Is there a way to purge the topic in Kafka?
  5. How does an offset expire for an Apache Kafka consumer group?
  6. How to commit manually with Kafka Stream?
  7. Increase the number of messages read by a Kafka consumer in a single poll
  8. Kafka how to read from __consumer_offsets topic
  9. How to change the number of replicas of a Kafka topic?
  10. Delete message after consuming it in KAFKA
  11. Understanding Kafka Topics and Partitions
  12. When to use RabbitMQ over Kafka? [closed]
  13. Delete topic in Kafka 0.8.1.1
  14. How to get Kafka offsets for structured query for manual and reliable offset management?
  15. Kafka Stream Suppress session-windowed-aggregation
  16. Use Confluent Hub without Confluent Platform installation
  17. Kafka 0.11 how to reset offsets
  18. offsetting an html anchor to adjust for fixed header [duplicate]
  19. Pagination using MySQL LIMIT, OFFSET
  20. Get position/offset of element relative to a parent container?
  21. SQL (ORACLE): ORDER BY and LIMIT [duplicate]
  22. How to write spark streaming DF to Kafka topic
  23. Writing Custom Kafka Serializer
  24. How to manually set group.id and commit kafka offsets in spark structured streaming?
  25. Connect to Kafka on host from Docker (ksqlDB)
  26. SyntaxError on “self.async” when running python kafka producer
  27. Can i add topics to my @kafkalistener at runtime
  28. How do I get the offset().top value of an element without using jQuery?
  29. How do you use offsetof() on a struct?
  30. Get the offset of a hidden element

Leave a Comment