Kafka how to read from __consumer_offsets topic

by

I came across this question when trying to also consume from the __consumer_offsets topic.
I managed to figure it out for different Kafka versions and thought I’d share what I’d found

For Kafka 0.8.2.x

Note: This uses Zookeeper connection

#Create consumer config
echo "exclude.internal.topics=false" > /tmp/consumer.config
#Consume all offsets
./kafka-console-consumer.sh --consumer.config /tmp/consumer.config \
--formatter "kafka.server.OffsetManager\$OffsetsMessageFormatter" \
--zookeeper localhost:2181 --topic __consumer_offsets --from-beginning

For Kafka 0.9.x.x and 0.10.x.x

#Create consumer config
echo "exclude.internal.topics=false" > /tmp/consumer.config
#Consume all offsets
./kafka-console-consumer.sh --new-consumer --consumer.config /tmp/consumer.config \
--formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" \
--bootstrap-server localhost:9092 --topic __consumer_offsets --from-beginning

For 0.11.x.x – 2.x

#Create consumer config
echo "exclude.internal.topics=false" > /tmp/consumer.config
#Consume all offsets
./kafka-console-consumer.sh --consumer.config /tmp/consumer.config \
--formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" \
--bootstrap-server localhost:9092 --topic __consumer_offsets --from-beginning

More Related Contents:

  1. Difference between session.timeout.ms and max.poll.interval.ms for Kafka >= 0.10.1
  2. Delete message after consuming it in KAFKA
  3. Understanding Kafka Topics and Partitions
  4. How to send final kafka-streams aggregation result of a time windowed KTable?
  5. Streamparse wordcount example
  6. What determines Kafka consumer offset?
  7. Is there a way to purge the topic in Kafka?
  8. How does an offset expire for an Apache Kafka consumer group?
  9. How to commit manually with Kafka Stream?
  10. Increase the number of messages read by a Kafka consumer in a single poll
  11. How to change the number of replicas of a Kafka topic?
  12. When to use RabbitMQ over Kafka? [closed]
  13. Delete topic in Kafka 0.8.1.1
  14. Kafka Stream Suppress session-windowed-aggregation
  15. Kafka Consumer CommitFailedException
  16. Can a Kafka consumer(0.8.2.2) read messages in batch
  17. Use Confluent Hub without Confluent Platform installation
  18. How to change start offset for topic?
  19. Kafka 0.11 how to reset offsets
  20. How Can we create a topic in Kafka from the IDE using API
  21. How to read records in JSON format from Kafka using Structured Streaming?
  22. Kafka SASL zookeeper authentication
  23. When does the Apache Kafka client throw a “Batch Expired” exception?
  24. In Apache Kafka why can’t there be more consumer instances than partitions?
  25. How to get kafka consume lag in java program
  26. Why I cannot connect to Kafka from outside?
  27. Handling exceptions in Kafka streams
  28. Is it possible to access message headers with Kafka Streams?
  29. How to get Kafka offsets for structured query for manual and reliable offset management?
  30. Pyspark 2.4.0, read avro from kafka with read stream – Python

Leave a Comment