编写一个简单的HadoopMapReduce程序,实现对一组文本文件中的单词进行计数统计。
时间: 2024-03-16 13:46:18 浏览: 14
好的,以下是一个简单的Hadoop MapReduce程序,实现对文本文件中的单词进行计数统计:
Mapper代码:
```
public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
// 将每行文本按照空格拆分成一个个单词
String[] words = value.toString().split(" ");
// 循环遍历每个单词,将其输出为<单词, 1>的形式
for (String w : words) {
word.set(w);
context.write(word, one);
}
}
}
```
Reducer代码:
```
public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
int sum = 0;
// 循环遍历每个单词,并将其出现的次数相加
for (IntWritable value : values) {
sum += value.get();
}
// 输出每个单词及其出现的次数
context.write(key, new IntWritable(sum));
}
}
```
Driver代码:
```
public class WordCount {
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(WordCountMapper.class);
job.setCombinerClass(WordCountReducer.class);
job.setReducerClass(WordCountReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
// 设置输入路径和输出路径
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
```
在执行时,需要指定输入路径和输出路径,例如:
```
hadoop jar wordcount.jar WordCount /input /output
```
其中,/input表示输入路径,/output表示输出路径。