spark streaming demo (java 1.8)
时间: 2023-04-27 08:01:54 浏览: 78
您好,以下是Spark Streaming的Java 1.8演示示例:
```java
import org.apache.spark.SparkConf;
import org.apache.spark.streaming.Durations;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaPairDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import scala.Tuple2;
import java.util.Arrays;
public class SparkStreamingDemo {
public static void main(String[] args) throws InterruptedException {
// 创建SparkConf对象
SparkConf conf = new SparkConf().setAppName("SparkStreamingDemo").setMaster("local[*]");
// 创建JavaStreamingContext对象
JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(1));
// 创建DStream
JavaDStream<String> lines = jssc.socketTextStream("localhost", 9999);
// 对DStream进行操作
JavaPairDStream<String, Integer> wordCounts = lines.flatMap(line -> Arrays.asList(line.split(" ")).iterator())
.mapToPair(word -> new Tuple2<>(word, 1))
.reduceByKey((a, b) -> a + b);
// 输出结果
wordCounts.print();
// 启动StreamingContext
jssc.start();
// 等待程序终止
jssc.awaitTermination();
}
}
```
这个示例程序从本地的9999端口接收数据流,将每行数据按空格分割成单词,然后统计每个单词出现的次数,并输出结果。您可以根据自己的需求修改程序中的参数和逻辑。