How can I send large messages with Kafka (over 15MB)?

by

You need to adjust three (or four) properties:

  • Consumer side:fetch.message.max.bytes – this will determine the largest size of a message that can be fetched by the consumer.
  • Broker side: replica.fetch.max.bytes – this will allow for the replicas in the brokers to send messages within the cluster and make sure the messages are replicated correctly. If this is too small, then the message will never be replicated, and therefore, the consumer will never see the message because the message will never be committed (fully replicated).
  • Broker side: message.max.bytes – this is the largest size of the message that can be received by the broker from a producer.
  • Broker side (per topic): max.message.bytes – this is the largest size of the message the broker will allow to be appended to the topic. This size is validated pre-compression. (Defaults to broker’s message.max.bytes.)

I found out the hard way about number 2 – you don’t get ANY exceptions, messages, or warnings from Kafka, so be sure to consider this when you are sending large messages.

More Related Contents:

  1. Connect to Kafka running in Docker
  2. How to create a Topic in Kafka through Java
  3. How Can we create a topic in Kafka from the IDE using API
  4. What determines Kafka consumer offset?
  5. Writing Custom Kafka Serializer
  6. Java, How to get number of messages in a topic in apache kafka
  7. How to Delete a topic in apache kafka [duplicate]
  8. When does the Apache Kafka client throw a “Batch Expired” exception?
  9. What is a simple, effective way to debug custom Kafka connectors?
  10. Is it possible to add partitions to an existing topic in Kafka 0.8.2
  11. How to get kafka consume lag in java program
  12. Handling exceptions in Kafka streams
  13. Is it possible to access message headers with Kafka Streams?
  14. Dynamically connecting a Kafka input stream to multiple output streams
  15. How to create Custom serializer in kafka?
  16. Kafka Consumer CommitFailedException
  17. Can a Kafka consumer(0.8.2.2) read messages in batch
  18. How can I delete a topic in Apache Kafka? [duplicate]
  19. How can I convert integer into float in Java?
  20. Print the stack trace of an exception
  21. Is this valid Java?
  22. How to get a complete stack trace of a running java program that is taking 100% cpu?
  23. Memory Fully utilized by Java ConcurrentHashMap (under Tomcat)
  24. Getting GPG Decryption To Work In Java (Bouncy Castle)
  25. problems with java3D lib configuration
  26. how to add checkbox and combobox in table cell?
  27. VariableDeclarationFragment node resolveBindind() returns null in eclipse/jdt/ast
  28. Calculating length in UTF-8 of Java String without actually encoding it
  29. When will System.currentTimeMillis() overflow?
  30. CXF JAXRS – How do I pass Date as QueryParam

Leave a Comment