MapReduce常用代码
时间: 2024-05-07 08:03:02 浏览: 17
MapReduce是一种分布式计算框架,主要用于大规模数据处理和分析。下面是一些常用的MapReduce代码示例:
1. Word Count示例:
Map函数:
```java
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
context.write(word, one);
}
}
```
Reduce函数:
```java
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
result.set(sum);
context.write(key, result);
}
```
2. Inverted Index示例:
Map函数:
```java
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
String[] words = line.split("\\s+");
for (String word : words) {
wordDoc.set(word + "@" + key.toString());
context.write(wordDoc, one);
}
}
```
Reduce函数:
```java
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
StringBuilder builder = new StringBuilder();
for (IntWritable value : values) {
builder.append(value.get() + ",");
}
String[] wordDoc = key.toString().split("@");
result.set(wordDoc[1] + ":" + builder.toString());
context.write(new Text(wordDoc[0]), result);
}
```
3. Join示例:
Map函数:
```java
public void map(Object key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
String[] words = line.split(",");
if (words.length == 2) {
context.write(new Text(words[0]), new Text("A:" + words[1]));
} else if (words.length == 3) {
context.write(new Text(words[1]), new Text("B:" + words[0] + "," + words[2]));
}
}
```
Reduce函数:
```java
public void reduce(Text key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
List<String> listA = new ArrayList<String>();
List<String> listB = new ArrayList<String>();
for (Text value : values) {
String[] words = value.toString().split(":");
if (words[0].equals("A")) {
listA.add(words[1]);
} else if (words[0].equals("B")) {
listB.add(words[1]);
}
}
for (String a : listA) {
for (String b : listB) {
context.write(key, new Text(a + "," + b));
}
}
}
```
这些代码示例只是MapReduce的冰山一角,它还有很多其他的应用场景和代码实现方式。