Java连接hdfs下载目录下文件成压缩包的工具类
时间: 2023-12-05 12:03:27 浏览: 77
你可以使用以下代码将HDFS目录下的文件打包成压缩包并下载:
```java
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.io.compress.CompressionCodec;
import org.apache.hadoop.io.compress.CompressionOutputStream;
import org.apache.hadoop.io.compress.GzipCodec;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
public class HdfsZipDownloadUtil {
public static void downloadAsZip(String hdfsPath, String localPath) throws Exception {
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf);
Path srcPath = new Path(hdfsPath);
ZipOutputStream zos = new ZipOutputStream(fs.create(new Path(localPath)));
// 使用压缩算法
CompressionCodec codec = new GzipCodec();
CompressionOutputStream cos = codec.createOutputStream(zos);
// 遍历目录下所有文件并压缩
for (Path path : FileUtil.stat2Paths(fs.listStatus(srcPath))) {
if (fs.isFile(path)) {
String fileName = path.getName();
ZipEntry zipEntry = new ZipEntry(fileName);
zos.putNextEntry(zipEntry);
InputStream inputStream = fs.open(path);
IOUtils.copyBytes(inputStream, cos, conf);
inputStream.close();
zos.closeEntry();
}
}
cos.finish();
cos.close();
zos.close();
}
}
```
其中,`hdfsPath`是要下载的HDFS目录路径,`localPath`是要保存到本地的文件路径。该方法会将目录下所有文件打包成 `.zip` 格式的压缩包并保存到本地。你可以按照需要修改压缩算法和保存的文件格式。
阅读全文