spark kafka write stream

Solutions on MaxInterview for spark kafka write stream by the best coders in the world

showing results for - "spark kafka write stream"
Damián
26 Aug 2018
1# Write key-value data from a DataFrame to a specific Kafka topic specified in an option
2ds = df \
3  .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)") \
4  .writeStream \
5  .format("kafka") \
6  .option("kafka.bootstrap.servers", "host1:port1,host2:port2") \
7  .option("topic", "topic1") \
8  .start()
9
10# Write key-value data from a DataFrame to Kafka using a topic specified in the data
11ds = df \
12  .selectExpr("topic", "CAST(key AS STRING)", "CAST(value AS STRING)") \
13  .writeStream \
14  .format("kafka") \
15  .option("kafka.bootstrap.servers", "host1:port1,host2:port2") \
16  .start()