Spark Integrates Kafka's Two Patterns
In development, we often use SparkStreaming to read and process data in kafka in real time. After version 1.3 of SparkStreaming, KafkaUtils provides two methods to create DStream:
Receiver reception: KafkaUtils.createDstream
There is a Receiver as a resident Task running in Executor waiting for data, but a Receiver is inefficient, need to open multiple, then manually merge data, and then process, very troublesome, and the Receiver machine hangs up, part of the data will be lost, need to open WAL...