How to commit manually with Kafka Stream?

by

Commits are handled by Streams internally and fully automatic, and thus there is usually no reason to commit manually. Note, that Streams handles this differently than consumer auto-commit — in fact, auto-commit is disabled for the internally used consumer and Streams manages commits “manually”. The reason is, that commits can only happen at certain points during processing to ensure no data can get lost (there a many internal dependencies with regard to updating state and flushing results).

For more frequent commits, you can reduce commit interval via StreamsConfig parameter commit.interval.ms.

Nevertheless, manual commits are possible indirectly, via low-level Processor API. You can use the context object that is provided via init() method to call context#commit(). Note, that this is only a “request to Streams” to commit as soon as possible — it’s not issuing a commit directly.

More Related Contents:

  1. How to send final kafka-streams aggregation result of a time windowed KTable?
  2. Kafka Stream Suppress session-windowed-aggregation
  3. Streamparse wordcount example
  4. Difference between session.timeout.ms and max.poll.interval.ms for Kafka >= 0.10.1
  5. Is there a way to purge the topic in Kafka?
  6. How does an offset expire for an Apache Kafka consumer group?
  7. Increase the number of messages read by a Kafka consumer in a single poll
  8. Kafka how to read from __consumer_offsets topic
  9. How to change the number of replicas of a Kafka topic?
  10. Delete message after consuming it in KAFKA
  11. Understanding Kafka Topics and Partitions
  12. When to use RabbitMQ over Kafka? [closed]
  13. Delete topic in Kafka 0.8.1.1
  14. Handling exceptions in Kafka streams
  15. Is it possible to access message headers with Kafka Streams?
  16. Dynamically connecting a Kafka input stream to multiple output streams
  17. Use Confluent Hub without Confluent Platform installation
  18. How to change start offset for topic?
  19. Kafka 0.11 how to reset offsets
  20. Connect to Kafka running in Docker
  21. How to install Kafka on Windows?
  22. Integrating Spark Structured Streaming with the Confluent Schema Registry
  23. How to create a Topic in Kafka through Java
  24. How to manually set group.id and commit kafka offsets in spark structured streaming?
  25. How to Delete a topic in apache kafka [duplicate]
  26. Is it possible to add partitions to an existing topic in Kafka 0.8.2
  27. Kafka access inside and outside docker [duplicate]
  28. How to get kafka consume lag in java program
  29. Can i add topics to my @kafkalistener at runtime
  30. How can I delete a topic in Apache Kafka? [duplicate]

Leave a Comment