Spark 2.0实战：打造实用机器学习系统

需积分: 10 13 浏览量更新于2024-07-17 收藏 17.87MB PDF 举报

"《机器学习与Spark（第二版）》是一本专注于实战应用的书籍，它将理论与实践相结合，引导读者在Spark 2.x环境中开发智能机器学习系统。作者Rajdeep Dua、Manpreet Singh Ghotra和Nick Pentreath通过实例和代码演示，帮助读者掌握如何利用Spark的MLlib库以及其它开源工具如机器学习和数据分析工具，构建实用的机器学习解决方案。书中强调了实际操作的重要性，虽然会涉及一些机器学习算法的基本概念和所需的数学基础，但主要目标是教会读者如何在实际项目中有效地运用这些技术。书中的内容涵盖了如何设计、实现和部署基于Spark的机器学习模型，包括数据预处理、特征工程、模型选择和评估等关键步骤。版权方面，该书享有2017年Packt Publishing的版权，所有复制、存储或任何形式的传输都必须得到出版商事先书面许可，除非在批评文章或评论中引用部分短语。尽管作者和出版社已尽力确保信息的准确性，但书中提供的内容并不构成任何形式的保证，无论是明示还是默示，都不承担因本书导致或声称导致的直接或间接损害的责任。此外，Packt Publishing努力确保书中提及的所有公司和产品的商标信息的准确性，但不能保证其绝对正确性。这本书旨在提供一个实践导向的学习资源，让读者在实际工作中能够快速上手并应用机器学习技术于Spark平台。"

Random forest regression
Gradient boosted tree regression
Improving model performance and tuning parameters
Transforming the target variable
Impact of training on log-transformed targets
Tuning model parameters
Creating training and testing sets to evaluate parameters
Splitting data for Decision tree
The impact of parameter settings for linear models
Iterations
Step size
L2 regularization
L1 regularization
Intercept
The impact of parameter settings for the decision tree
Tree depth
Maximum bins
The impact of parameter settings for the Gradient Boosted Trees
Iterations
MaxBins
Summary
8.  Building a Clustering Model with Spark
Types of clustering models
k-means clustering
Initialization methods
Mixture models
Hierarchical clustering
Extracting the right features from your data
Extracting features from the MovieLens dataset
K-means - training a clustering model
Training a clustering model on the MovieLens dataset
K-means - interpreting cluster predictions on the MovieLens dataset
Interpreting the movie clusters
Interpreting the movie clusters
K-means - evaluating the performance of clustering models
Internal evaluation metrics
External evaluation metrics
Computing performance metrics on the MovieLens dataset
Effect of iterations on WSSSE
Bisecting KMeans
Bisecting K-means - training a clustering model
WSSSE and iterations
Gaussian Mixture Model
Clustering using GMM
Plotting the user and item data with GMM clustering
GMM - effect of iterations on cluster boundaries
Summary
16

Preface

In recent years, the volume of data being collected, stored, and analyzed has

exploded, in particular in relation to activity on the Web and mobile devices, as

well as data from the physical world collected via sensor networks. While large-

scale data storage, processing, analysis, and modeling were previously the domain

of the largest institutions, such as Google, Yahoo!, Facebook, Twitter, and

Salesforce, increasingly, many organizations are being faced with the challenge of

how to handle a massive amount of data.

When faced with this quantity of data and the common requirement to utilize it in

real time, human-powered systems quickly become infeasible. This has led to a rise

in so-called big data and machine learning systems that learn from this data to

make automated decisions.

In answer to the challenge of dealing with ever larger-scale data without any

prohibitive cost, new open source technologies emerged at companies such as

Google, Yahoo!, Amazon, and Facebook, which aimed at making it easier to

handle massive data volumes by distributing data storage and computation across a

cluster of computers.

The most widespread of these is Apache Hadoop, which made it significantly

easier and cheaper to both store large amounts of data (via the Hadoop Distributed

File System, or HDFS) and run computations on this data (via Hadoop

MapReduce, a framework to perform computation tasks in parallel across many

nodes in a computer cluster).

However, MapReduce has some important shortcomings, including high overheads

to launch each job and reliance on storing intermediate data and results of the

computation to disk, both of which make Hadoop relatively ill-suited for use cases

of an iterative or low-latency nature. Apache Spark is a new framework for

distributed computing that is designed from the ground up to be optimized for low-

latency tasks and to store intermediate data and results in memory, thus addressing

some of the major drawbacks of the Hadoop framework. Spark provides a clean,

functional, and easy-to-understand API to write applications, and is fully

compatible with the Hadoop ecosystem.

Furthermore, Spark provides native APIs in Scala, Java, Python, and R. The Scala

and Python APIs allow all the benefits of the Scala or Python language,

respectively, to be used directly in Spark applications, including using the relevant

剩余651页未读，继续阅读

风语飘摇

粉丝: 88
资源: 16

Spark 2.0实战：打造实用机器学习系统

Mastering Machine Learning with Spark 2.X 无水印pdf

Machine Learning with Spark - Second Edition

Machine Learning with Spark 2nd Edition

hands-on machine learning with scikit-learn, keras & tensorflow

hands-on machine learning with scikit-learn, keras, andtensorflow

hands-on machine learning with scikit-learn, keras, and tensorflow pdf

pdf下载hands-on machine learning with scikit-learn, keras & tensorflow, 2nd

Pattern Recognition and Machine Learning-01-Preface

machine learning with pytorch and scikit-learn pdf

The Non-IID Data Quagmire of Decentralized Machine Learning

最新资源