MapReduce编程实现文件合并和去重操作

假设有两个文件file1.txt和file2.txt，需要将它们合并并去重，可以使用MapReduce编程实现。

1. Map阶段

将每个文件的每一行作为key，value为空，发射到Reducer。

2. Reduce阶段

将所有相同的key合并，并将value设为1，最终输出所有不重复的行。

具体的代码实现如下：

Mapper代码：

public class MergeMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    private Text word = new Text();
    private final static IntWritable one = new IntWritable(1);

    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String line = value.toString();
        word.set(line);
        context.write(word, one);
    }
}

Reducer代码：

public class MergeReducer extends Reducer<Text, IntWritable, Text, NullWritable> {
    public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
        context.write(key, NullWritable.get());
    }
}

Driver代码：

public class MergeDriver {
    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "Merge and Deduplication");

        job.setJarByClass(MergeDriver.class);
        job.setMapperClass(MergeMapper.class);
        job.setReducerClass(MergeReducer.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileInputFormat.addInputPath(job, new Path(args[1]));
        FileOutputFormat.setOutputPath(job, new Path(args[2]));

        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

以上代码实现了将两个文件合并并去重的操作，通过传入不同的输入和输出路径，可以实现不同的文件合并和去重操作。