spark的example

1. Word Count
This example counts the occurrence of each word in a text file using Spark.

val textFile = sc.textFile("hdfs://...")
val counts = textFile.flatMap(line => line.split(" "))
                 .map(word => (word, 1))
                 .reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")

2. Pi Estimation
This example uses a Monte Carlo method to estimate the value of pi using Spark.

val NUM_SAMPLES = 1000000
val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
  val x = Math.random()
  val y = Math.random()
  if (x*x + y*y < 1) 1 else 0
}.reduce(_ + _)
println("Pi is roughly " + 4.0 * count / NUM_SAMPLES)

3. PageRank
This example calculates the PageRank of web pages using Spark.

val links = sc.parallelize(Seq((1, Seq(2, 3)), (2, Seq(1)), (3, Seq(2))))
var ranks = links.mapValues(v => 1.0)
for (i <- 1