掌握Spark 2.0入门指南：Scala与Python实战

需积分: 10 113 浏览量更新于2024-07-20 收藏 20.46MB PDF 举报

《Apache Spark 2.0入门指南》是一本专为初学者设计的教程，旨在帮助读者掌握Apache Spark 2.0这一广泛应用的大规模分布式数据处理引擎。Spark以其快速性能和广泛的功能深受开发者和数据科学家的青睐，特别是通过SparkR（在Spark框架中的“R版本”）扩展了对R用户的处理能力。本书首先从Spark 2.0的基础知识出发，涵盖了核心数据处理框架、API安装以及应用开发环境的搭建。通过实际案例介绍，读者将深入了解Spark编程模型，然后深入学习如何使用Scala和Python进行Spark SQL编程，特别是DataFrame的使用。 SparkR部分让读者接触到R语言在Spark环境下的应用，这对于那些习惯于R语言分析的用户来说是一个重要的补充。接下来，书中介绍了如何利用Python与Spark结合进行数据处理、图表绘制和可视化。此外，作者还将带领读者探索Spark的实时流处理、机器学习和图计算库，这些都是现代数据处理不可或缺的技术。最后一章将综合运用前面学到的所有技能，指导读者开发一个完整的Spark应用程序，从而实现在实际项目中的应用。无论你是Scala、Python还是R的开发者，或者对大数据处理有兴趣的数据科学家，这本书都将提供一个全面且实用的Spark 2.0学习路径。需要注意的是，本书版权受到保护，未经版权所有者Packt Publishing许可，不得复制、存储或传播书中的任何内容。阅读本书，你将收获以下关键技能： 1. **理解Spark 2.0基础**：掌握Spark的基本概念和编程模型，包括在Scala和Python中的实现。 2. **使用Spark SQL和DataFrame**：学习如何在Scala和Python中操作Spark的数据处理和SQL查询。 3. **SparkR入门**：了解R语言在Spark环境中的应用和编程方法。 4. **Python数据处理**：学会结合Python进行高效的数据处理和可视化。 5. **实时流处理**：学会利用Scala和Python进行Spark的实时数据流处理。 6. **机器学习实战**：使用Spark进行基于Scala和Python的机器学习项目。 7. **图计算入门**：接触并了解如何使用Spark进行图数据处理。《Apache Spark 2.0 for Beginners》是一本全面的指南，旨在帮助你快速上手Spark，开启大规模数据处理的新征程。

Preface

[ 4 ]

Who this book is for

If you are an application developer, data scientist, or big data solutions architect who is

interested in combining the data processing power of Spark with R, and consolidating data

processing, stream processing, machine learning, and graph processing into one unified and

highly interoperable framework with a uniform API using Scala or Python, this book is for

you.

Conventions

In this book, you will find a number of text styles that distinguish between different kinds

of information. Here are some examples of these styles and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions,

pathnames, dummy URLs, user input, and Twitter handles are shown as follows: " It is a

good idea to customize this property spark.driver.memory to have a higher value."

A block of code is set as follows:

Python 3.5.0 (v3.5.0:374f501f4567, Sep 12 2015, 11:00:19)

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin

Any command-line input or output is written as follows:

$ python

Python 3.5.0 (v3.5.0:374f501f4567, Sep 12 2015, 11:00:19)

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>>

New terms and important words are shown in bold. Words that you see on the screen, for

example, in menus or dialog boxes, appear in the text like this: "The shortcuts in this book

are based on the Mac OS X 10.5+ scheme."

Preface
[ 6 ]
Once the file is downloaded, please make sure that you unzip or extract the folder using the
latest version of:
WinRAR / 7-Zip for Windows
Zipeg / iZip / UnRarX for Mac
7-Zip / PeaZip for Linux
The code bundle for the book is also hosted on GitHub at h t t p s : / / g i t h u b . c o m / P a c k t P u b l
i s h i n g / A p a c h e - S p a r k - 2 - f o r - B e g i n n e r s. We also have other code bundles from our rich
catalog of books and videos available at h t t p s : / / g i t h u b . c o m / P a c k t P u b l i s h i n g /. Check
them out!
Downloading the color images of this book
We also provide you with a PDF file that has color images of the screenshots/diagrams used
in this book. The color images will help you better understand the changes in the output.
You can download this file from h t t p : / / w w w . p a c k t p u b . c o m / s i t e s / d e f a u l t / f i l e s / d o w n l
o a d s / A p a c h e S p a r k 2 f o r B e g i n n e r s _ C o l o r I m a g e s . p d f.
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes do
happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-
we would be grateful if you could report this to us. By doing so, you can save other readers
from frustration and help us improve subsequent versions of this book. If you find any
errata, please report them by visiting h t t p : / / w w w . p a c k t p u b . c o m / s u b m i t - e r r a t a, selecting
your book, clicking on the Errata Submission Form link, and entering the details of your
errata. Once your errata are verified, your submission will be accepted and the errata will
be uploaded to our website or added to any list of existing errata under the Errata section of
that title.
To view the previously submitted errata, go to h t t p s : / / w w w . p a c k t p u b . c o m / b o o k s / c o n t e n
t / s u p p o r t and enter the name of the book in the search field. The required information will
appear under the Errata section.

Spark Fundamentals

Data is one of the most important assets of any organization. The scale at which data is

being collected and used in organizations is growing beyond imagination. The speed at

which data is being ingested, the variety of the data types in use, and the amount of data

that is being processed and stored are breaking all-time records every moment. It is very

common these days, even in small-scale organizations, that data is growing from gigabytes

to terabytes to petabytes. For the same reason, the processing needs are also growing that

ask for capability to process data at rest as well as data on the move.

Take any organization; its success depends on the decisions made by its leaders and for

making sound decisions, you need the backing of good data and the information generated

by processing the data. This poses a big challenge on how to process the data in a timely

and cost-effective manner so that right decisions can be made. Data processing techniques

have evolved since the early days of computers. Countless data processing products and

frameworks came into the market and disappeared over these years. Most of these data

processing products and frameworks were not general purpose in nature. Most of the

organizations relied on their own bespoke applications for their data processing needs, in a

silo way, or in conjunction with specific products.

Large-scale Internet applications, popularly known as Internet of Things (IoT)

applications, heralded the common need to have open frameworks to process huge

amounts of data ingested at great speed dealing with various types of data. Large-scale web

sites, media streaming applications, and the huge batch processing needs of organizations

made the need even more relevant. The open source community is also growing

considerably along with the growth of the Internet, delivering production quality software

supported by reputed software companies. A huge number of companies started using

open source software and started deploying them in their production environments.

剩余321页未读，继续阅读

jamesxu2016

粉丝: 1
资源: 8

掌握Spark 2.0入门指南：Scala与Python实战

Spark 2.0 for Beginners 无水印pdf

Spark 2.0 for Beginners mobi

Spark 2.0 for Beginners(PACKT,2016)

spark2.0 for Begginners

Apache Spark 2 for Beginners [2016]

Spark 2.0初学者指南：分布式数据处理

Spark 2.0入门：大数据处理与实战

office2john的源文件

青海省各市、县区及街镇SVG图

(源码)基于JavaNIO框架的网络IO处理系统.zip

最新资源