Scala Java Python
Scala Java Python
Resilient Distributed Datasets (RDDs)
Spark revolves around the concept of a resilient distributed dataset (RDD), which is a fault-
tolerant collection of elements that can be operated on in parallel. There are two ways to create
RDDs: parallelizing an existing collection in your driver program, or referencing a dataset in an
external storage system, such as a shared filesystem, HDFS, HBase, or any data source
offering a Hadoop InputFormat.
Parallelized Collections
Parallelized collections are created by calling
JavaSparkContext
’s
parallelize
method on an
existing
Collection
in your driver program. The elements of the collection are copied to form a
distributed dataset that can be operated on in parallel. For example, here is how to create a
parallelized collection holding the numbers 1 to 5:
List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);
JavaRDD<Integer> distData = sc.parallelize(data);
Once created, the distributed dataset (
distData
) can be operated on in parallel. For example,
we might call
distData.reduce((a, b) -> a + b)
to add up the elements of the list. We
describe operations on distributed datasets later on.
Note: In this guide, we’ll often use the concise Java 8 lambda syntax to specify Java functions,
but in older versions of Java you can implement the interfaces in the
org.apache.spark.api.java.function package. We describe passing functions to Spark in more
detail below.
One important parameter for parallel collections is the number of partitions to cut the dataset
into. Spark will run one task for each partition of the cluster. Typically you want 2-4 partitions for
each CPU in your cluster. Normally, Spark tries to set the number of partitions automatically
based on your cluster. However, you can also set it manually by passing it as a second
parameter to
parallelize
(e.g.
sc.parallelize(data, 10)
). Note: some places in the code
use the term slices (a synonym for partitions) to maintain backward compatibility.
External Datasets
Spark can create distributed datasets from any storage source supported by Hadoop, including
your local file system, HDFS, Cassandra, HBase, Amazon S3, etc. Spark supports text files,
SequenceFiles, and any other Hadoop InputFormat.
剩余19页未读，继续阅读
wang_18369956863
- 粉丝: 0
- 资源: 1
我的内容管理
展开
最新资源
- zlib-1.2.12压缩包解析与技术要点
- 微信小程序滑动选项卡源码模版发布
- Unity虚拟人物唇同步插件Oculus Lipsync介绍
- Nginx 1.18.0版本WinSW自动安装与管理指南
- Java Swing和JDBC实现的ATM系统源码解析
- 掌握Spark Streaming与Maven集成的分布式大数据处理
- 深入学习推荐系统：教程、案例与项目实践
- Web开发者必备的取色工具软件介绍
- C语言实现李春葆数据结构实验程序
- 超市管理系统开发：asp+SQL Server 2005实战
- Redis伪集群搭建教程与实践
- 掌握网络活动细节：Wireshark v3.6.3网络嗅探工具详解
- 全面掌握美赛：建模、分析与编程实现教程
- Java图书馆系统完整项目源码及SQL文件解析
- PCtoLCD2002软件：高效图片和字符取模转换
- Java开发的体育赛事在线购票系统源码分析
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
