mapreduce api
时间: 2023-11-24 20:53:20 浏览: 113
MapReduce是一种用于大规模数据处理的编程模型和算法。它将大规模数据集分成小的数据块,然后在集群中的多台计算机上并行处理这些数据块。MapReduce API是一种用于实现MapReduce算法的编程接口。以下是一个简单的MapReduce API的例子:
```java
public class WordCount {
public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1); private Text word = new Text();
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
context.write(word, one);
}
}
}
public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
context.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = new Job(conf, "wordcount");
job.setJarByClass(WordCount.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
}
}
```
这个例子是一个简单的WordCount程序,它将输入文件中的单词计数,并将结果写入输出文件。Map函数将输入文件中的每一行拆分成单词,并将每个单词映射到一个键值对,其中键是单词,值是1。Reduce函数将相同键的值相加,并将结果写入输出文件。