How to write spark streaming DF to Kafka topic
Yes, unfortunately Spark (1.x, 2.x) doesn’t make it straight-forward how to write to Kafka in an efficient manner. I’d suggest the following approach: Use (and re-use) one KafkaProducer instance per executor process/JVM. Here’s the high-level setup for this approach: First, you must “wrap” Kafka’s KafkaProducer because, as you mentioned, it is not serializable. Wrapping it … Read more