没有合适的资源？快使用搜索试试~ 我知道了~

首页Spark中的压缩编解码的最佳实践

Spark中的压缩编解码的最佳实践

Spark

压缩编解码

需积分: 18 12 下载量 148 浏览量更新于2023-03-16 评论收藏 1.38MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

试读

36页

如何在数据压缩率和编解码效率之间找到平衡，该分享介绍Intel大数据团队在为Spark实现 ISA-L(iGZIP), LZ4-IPP, ZLIB-IPP 和 ZSTD等针对IA硬件平台优化的算法支持，并使用基准测试集(TPC-DS/HiBench)对这些压缩编解码在Spark上的性能表现进行的详细分析和对比

资源详情

资源评论

资源推荐

SSG / DPD / BDT

Sophia Sun (sophia.sun@intel.com)

Qi Xie (qi.xie@intel.com)

Hao Cheng (hao.cheng@intel.com)

Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.

*Other names and brands may be claimed as the property of others

Outlines

Background

Micro-benchmark Result

TPC-DS* benchmark Result

HiBench* benchmark Result

Intel® Compression Codec Architecture

*Other names and brands may be claimed as the property of others.

Compression Needs and Trade-offs in Spark*

• Why we need data compression?

• Save storage space.

• In data intensive workload, disk and network may as bottleneck. Compressing data can

speed up the I/O operations and data transfer across network.

• Spark Shuffle process has huge I/O pressure as it has to often "spill out" intermediate

data to local disks before advancing from Map stage to Reduce stage.

• Data Compression trade-offs

• Although compression could reduce I/O and network pressure, CPU utilization

generally increases as data must be decompressed before the files can be processed.

*Other names and brands may be claimed as the property of others.

Compression Codecs Info

Codecs

Supported

levels

Default

level

Degree of

Compression

speed

CPU Usage

Comments

ISA

-L(igzip

)

(0~1)

Medium

Medium~High

Based on Intel® ISA

-L

optimization

Zlib

-ipp

(1~9)

Best

balance(near

to 6)

High

Slow

High

Based on Intel® IPP

library optimization

Lz4

-ipp

N/A

Medium

Fast

Low

Based on Intel® IPP

library optimization

Lz4

Lz4 fast

Lz4

fast

Low

Medium

Fast

Low

Medium

Open source codec

snappy

N/A

Low

Fast

Low

Open source codec

Zlib

/gzip

(1~9)

Best

balance(near

to 6)

High

Slow

High

Open source codec

zstd

1~22

High

Medium

Medium~High

Open source codec

Map

InMemory

Buffer

Input Splits

Partition and Sort

Map O/P

Fetch

Reduce I/P

Other Map

O/P

Reduce

Merge and Sort

Output

Map Reduce

Other

Reducers

I/P Compressed

Mapper Decompression

Mapper O/P Compressed

Reducer Decompression

Reducer O/P

Compressed

Data Compression Pipeline in Spark

剩余35页未读，继续阅读

Erjin_Ren

粉丝: 13
资源: 17

上传资源快速赚钱

我的内容管理收起

我的资源快来上传第一个资源

我的收益

登录查看自己的收益

我的积分登录查看自己的积分

我的C币登录后查看C币余额

我的收藏

我的下载

下载帮助

会员权益专享

Spark中的压缩编解码的最佳实践

评论0

会员权益专享

最新资源

Spark中的压缩编解码的最佳实践

评论0

hibench大数据性能测试工具

Spark的shuffle调优

Spark SQL最佳实践.pdf

spark实践 | spark 大数据处理最佳实践

如何设置spark不压缩

大数据最佳实践-hive on spark

大数据最佳实践-spark structstreaming

如何在sparkshell里面设置spark不压缩

spark(61) -- sparkmllib -- 基于sparkmllib的数据清洗项目实践

spark2中使用龙芯的lzs压缩算法

spark程序中如何使用龙芯的压缩裤、

spark大数据分析与实战：spark streaming编程初级实践spark streaming编程初级实践

Spark初级编程实践

spark机器学习库mllib编程实践

spark sql 物化视图原理与实践

spark streaming编程初级实践

国外spark实践视频

spark 读取压缩文件

spark sql 编程实践实验建议

实验spark安装和编程实践

会员权益专享

最新资源