利用Spark构建可扩展的机器学习应用驱动商业

需积分: 4 159 浏览量更新于2024-07-18 收藏 6.68MB PDF 举报

《使用Spark进行机器学习》是一本由Nick Pentreath编著的专业书籍，它专注于如何利用Apache Spark构建可扩展的机器学习应用程序，以推动现代数据驱动的企业决策。这本书适合对大数据处理和机器学习有深入理解的读者，特别是那些希望在分布式计算框架上提升其算法性能和业务应用的人。 Spark作为一种强大的开源工具，提供了分布式计算、内存计算和流处理的能力，使得大规模机器学习任务的处理变得高效且易于管理。本书覆盖了从基础概念到实践案例的广泛内容，包括但不限于： 1. **Spark概述**：介绍了Spark的架构、核心组件（如Spark Core、Spark SQL、MLlib等）以及其在大数据处理中的优势。 2. **数据预处理**：讲解如何在Spark上清洗、转换和整合数据，为机器学习做好准备，涉及数据清洗、特征工程、数据加载和存储等。 3. **机器学习算法**：涵盖了各种监督学习（如回归、分类、聚类）、无监督学习（如推荐系统、异常检测）以及深度学习的基本原理和实现方法。Spark MLlib库提供了丰富的模型和工具。 4. **分布式计算与优化**：阐述了如何利用Spark的并行计算能力，通过RDD（弹性分布式数据集）和DAG（有向无环图）模型提高机器学习任务的执行效率。 5. **实战项目**：书中包含多个实际项目的案例分析，通过这些示例，读者可以学习如何将所学理论应用于实际场景，解决商业问题。 6. **最佳实践与部署**：讨论了如何部署和维护Spark机器学习应用，包括集群配置、性能调优以及如何与Hadoop生态系统无缝集成。 7. **版权与使用限制**：强调了版权信息，指出未经许可，不得复制或传播书中的内容，且出版商对书中信息的准确性不承担任何责任。《使用Spark进行机器学习》不仅提供理论知识，还提供了实用的工具和技术，是数据科学家和工程师提升Spark技能，探索大数据处理与机器学习结合的宝贵资源。通过阅读和实践，读者能够深入了解如何利用Spark平台在当今快速变化的商业环境中推动数据分析和智能化决策。

Preface

In recent years, the volume of data being collected, stored, and analyzed has

exploded, in particular in relation to the activity on the Web and mobile devices, as

well as data from the physical world collected via sensor networks. While previously

large-scale data storage, processing, analysis, and modeling was the domain of the

largest institutions such as Google, Yahoo!, Facebook, and Twitter, increasingly,

many organizations are being faced with the challenge of how to handle a massive

amount of data.

When faced with this quantity of data and the common requirement to utilize it in

real time, human-powered systems quickly become infeasible. This has led to a rise

in the so-called big data and machine learning systems that learn from this data to

make automated decisions.

In answer to the challenge of dealing with ever larger-scale data without any

prohibitive cost, new open source technologies emerged at companies such as

Google, Yahoo!, Amazon, and Facebook, which aimed at making it easier to handle

massive data volumes by distributing data storage and computation across a cluster

of computers.

The most widespread of these is Apache Hadoop, which made it signicantly easier

and cheaper to both store large amounts of data (via the Hadoop Distributed File

System, or HDFS) and run computations on this data (via Hadoop MapReduce,

a framework to perform computation tasks in parallel across many nodes in a

computer cluster).

https://www.iteblog.com

Preface

[ 2 ]

However, MapReduce has some important shortcomings, including high overheads

to launch each job and reliance on storing intermediate data and results of the

computation to disk, both of which make Hadoop relatively ill-suited for use cases of

an iterative or low-latency nature. Apache Spark is a new framework for distributed

computing that is designed from the ground up to be optimized for low-latency

tasks and to store intermediate data and results in memory, thus addressing some of

the major drawbacks of the Hadoop framework. Spark provides a clean, functional,

and easy-to-understand API to write applications and is fully compatible with the

Hadoop ecosystem.

Furthermore, Spark provides native APIs in Scala, Java, and Python. The Scala and

Python APIs allow all the benets of the Scala or Python language, respectively,

to be used directly in Spark applications, including using the relevant interpreter

for real-time, interactive exploration. Spark itself now provides a toolkit (called

MLlib) of distributed machine learning and data mining models that is under heavy

development and already contains high-quality, scalable, and efcient algorithms for

many common machine learning tasks, some of which we will delve into in this book.

Applying machine learning techniques to massive datasets is challenging, primarily

because most well-known machine learning algorithms are not designed for parallel

architectures. In many cases, designing such algorithms is not an easy task. The

nature of machine learning models is generally iterative, hence the strong appeal

of Spark for this use case. While there are many competing frameworks for parallel

computing, Spark is one of the few that combines speed, scalability, in-memory

processing, and fault tolerance with ease of programming and a exible, expressive,

and powerful API design.

Throughout this book, we will focus on real-world applications of machine learning

technology. While we may briey delve into some theoretical aspects of machine

learning algorithms, the book will generally take a practical, applied approach with

a focus on using examples and code to illustrate how to effectively use the features

of Spark and MLlib, as well as other well-known and freely available packages for

machine learning and data analysis, to create a useful machine learning system.

What this book covers

Chapter 1, Getting Up and Running with Spark, shows how to install and set up a local

development environment for the Spark framework as well as how to create a Spark

cluster in the cloud using Amazon EC2. The Spark programming model and API will

be introduced, and a simple Spark application will be created using each of Scala,

Java, and Python.

https://www.iteblog.com

Preface

[ 3 ]

Chapter 2, Designing a Machine Learning System, presents an example of a real-world

use case for a machine learning system. We will design a high-level architecture for

an intelligent system in Spark based on this illustrative use case.

Chapter 3, Obtaining, Processing, and Preparing Data with Spark, details how to go about

obtaining data for use in a machine learning system, in particular from various freely

and publicly available sources. We will learn how to process, clean, and transform

the raw data into features that may be used in machine learning models, using

available tools, libraries, and Spark's functionality.

Chapter 4, Building a Recommendation Engine with Spark, deals with creating a

recommendation model based on the collaborative ltering approach. This model

will be used to recommend items to a given user as well as create lists of items

that are similar to a given item. Standard metrics to evaluate the performance of a

recommendation model will be covered here.

Chapter 5, Building a Classication Model with Spark, details how to create a model

for binary classication as well as how to utilize standard performance-evaluation

metrics for classication tasks.

Chapter 6, Building a Regression Model with Spark, shows how to create a model

for regression, extending the classication model created in Chapter 5, Building a

Classication Model with Spark. Evaluation metrics for the performance of regression

models will be detailed here.

Chapter 7, Building a Clustering Model with Spark, explores how to create a clustering

model as well as how to use related evaluation methodologies. You will learn how to

analyze and visualize the clusters generated.

Chapter 8, Dimensionality Reduction with Spark, takes us through methods to extract

the underlying structure from and reduce the dimensionality of our data. You will

learn some common dimensionality-reduction techniques and how to apply and

analyze them, as well as how to use the resulting data representation as input to

another machine learning model.

Chapter 9, Advanced Text Processing with Spark, introduces approaches to deal with

large-scale text data, including techniques for feature extraction from text and

dealing with the very high-dimensional features typical in text data.

Chapter 10, Real-time Machine Learning with Spark Streaming, provides an overview

of Spark Streaming and how it ts in with the online and incremental learning

approaches to apply machine learning on data streams.

https://www.iteblog.com

剩余337页未读，继续阅读

escaiyi

粉丝: 1
资源: 9

利用Spark构建可扩展的机器学习应用驱动商业

Machine Learning with Spark - Second Edition

Large Scale Machine Learning with Spark-Packt Publishing(2016).pdf

Machine Learning with Spark(PACKT,2015)

spark大数据编程头歌

pyspark学习推荐书籍

python大数据处理的书籍推荐

Spark和NLP技术参考文献

给我一个用Scala编写的复杂一点的和药相关的spark实例，包含代码和数据获取网址

spark hadoop

sqlserver 18

最新资源