hadoop运行scala

要在Hadoop上运行Scala程序，有两种方法：

1. 使用Hadoop Streaming

Hadoop Streaming是一种通用的方式，可以用来在Hadoop上运行任何语言的程序，包括Scala。您可以将Scala程序打包为一个可执行的JAR文件，并使用Hadoop Streaming来运行它。以下是一个示例：

$HADOOP_HOME/bin/hadoop jar /path/to/hadoop-streaming.jar \
    -input /path/to/input \
    -output /path/to/output \
    -mapper "scala my.package.MyMapper" \
    -reducer "scala my.package.MyReducer"

2. 使用Hadoop Scala API

Hadoop Scala API是Hadoop提供的Scala API，可以用来编写Hadoop作业。您可以使用它来编写Scala程序，并在Hadoop集群上运行它们。以下是一个示例：

import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.fs.Path
import org.apache.hadoop.io.IntWritable
import org.apache.hadoop.io.Text
import org.apache.hadoop.mapreduce.Job
import org.apache.hadoop.mapreduce.Mapper
import org.apache.hadoop.mapreduce.Reducer
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

object WordCount {
  def main(args: Array[String]) {
    val job = Job.getInstance()
    job.setJarByClass(WordCount.getClass())
    job.setJobName("wordcount")

    FileInputFormat.addInputPath(job, new Path(args(0)))
    FileOutputFormat.setOutputPath(job, new Path(args(1)))

    job.setMapperClass(classOf[TokenizerMapper])
    job.setCombinerClass(classOf[IntSumReducer])
    job.setReducerClass(classOf[IntSumReducer])

    job.setOutputKeyClass(classOf[Text])
    job.setOutputValueClass(classOf[IntWritable])

    val success = job.waitForCompletion(true)
    System.exit(if (success) 0 else 1)
  }
}

class TokenizerMapper extends Mapper[Object, Text, Text, IntWritable] {
  private val one = new IntWritable(1)
  private val word = new Text()

  override def map(key: Object, value: Text, context: Mapper[Object, Text, Text, IntWritable]#Context) {
    value.toString().split("\\s+").foreach { w =>
      word.set(w)
      context.write(word, one)
    }
  }
}

class IntSumReducer extends Reducer[Text, IntWritable, Text, IntWritable] {
  private val result = new IntWritable()

  override def reduce(key: Text, values: java.lang.Iterable[IntWritable], context: Reducer[Text, IntWritable, Text, IntWritable]#Context) {
    val sum = values.asScala.foldLeft(0)(_ + _.get())
    result.set(sum)
    context.write(key, result)
  }
}

要编译和打包此程序，您需要使用sbt或Maven等构建工具。在此之后，您可以将JAR文件提交到Hadoop集群并运行它。例如：

$HADOOP_HOME/bin/hadoop jar /path/to/wordcount.jar /path/to/input /path/to/output