使用java、scala、python三种语言实现spark rdd的reduce算子的使用

Java:

JavaRDD<Integer> numbers = sc.parallelize(Arrays.asList(1, 2, 3, 4, 5));
int sum = numbers.reduce((a, b) -> a + b);
System.out.println(sum);

Scala:

val numbers = sc.parallelize(Seq(1, 2, 3, 4, 5))
val sum = numbers.reduce(_ + _)
println(sum)

Python:

numbers = sc.parallelize([1, 2, 3, 4, 5])
sum = numbers.reduce(lambda a, b: a + b)
print(sum)

相关问题

请用java,scala,python三种语言实现spark的reducebykey案例

Java:

JavaPairRDD<String, Integer> rdd = ...;
JavaPairRDD<String, Integer> reducedRdd = rdd.reduceByKey((a, b) -> a + b);

Scala:

val rdd: RDD[(String, Int)] = ...
val reducedRdd = rdd.reduceByKey(_ + _)

Python:

rdd = sc.parallelize([('a', 1), ('a', 2), ('b', 1), ('b', 2)])
reducedRdd = rdd.reduceByKey(lambda a, b: a + b)

使用Spark RDD来分析movies.csv文件,统计每种类型的电影有多少部，用scala编程语言

好的，以下是使用Spark RDD来分析movies.csv文件，统计每种类型的电影有多少部的Scala代码：

import org.apache.spark.{SparkConf, SparkContext}

val conf = new SparkConf().setAppName("movie analysis")
val sc = new SparkContext(conf)

val data = sc.textFile("movies.csv")
val header = data.first()
val genres = data.filter(line => line != header)
                 .flatMap(line => line.split(",")(2).split("\\|"))
                 .distinct()
                 .collect()

val movieCountByGenre = data.filter(line => line != header)
                            .flatMap(line => line.split(",")(2).split("\\|").map((_, 1)))
                            .reduceByKey(_ + _)
                            .collect()

println("Number of movies by genre:")
for (genre <- genres) {
  val count = movieCountByGenre.find(_._1 == genre).map(_._2).getOrElse(0)
  println(s"$genre: $count")
}

这段代码与Python代码的思路相同，只是使用了Scala语言的语法。