掌握下一代Hadoop数据处理平台：进阶指南

5星 · 超过95%的资源需积分: 9 156 浏览量更新于2024-07-22 2 收藏 4.44MB PDF 举报

"《Packt.Mastering.Hadoop.2014》是一本由Sandeep Karanth编著的专业书籍，旨在帮助读者深入了解和掌握Hadoop这一数据处理平台的高级特性。本书针对的是已经掌握了基础Hadoop知识的学习者，它引导读者进入Hadoop的下一代技术栈，探索其更深层次的应用和优化策略。在书中，作者深入剖析了Hadoop的核心组件，如HDFS（Hadoop分布式文件系统）和MapReduce，以及后来出现的其他技术扩展，如YARN（Yet Another Resource Negotiator）、Hive、Pig和Spark等。读者可以在这里学习如何设计和实现高效的数据处理解决方案，包括大数据的存储、处理、分析和可视化。版权方面，所有权利受到Packt Publishing严格的保护，未经出版商事先书面许可，不得进行任何形式的复制、存储或传输。尽管作者和出版社尽力确保信息的准确性，但本书不提供任何形式的保证，无论是明示的还是暗示的，对于因使用本书信息导致的直接或间接损失，作者和出版社及其经销商均不承担法律责任。出版时间为2014年12月，生产参考号为1221214。《Mastering Hadoop 2014》这本书强调实践性和实用性，适合希望在大数据领域进一步提升技能的开发者、数据分析师和系统管理员阅读，通过不断学习，读者能够紧跟Hadoop技术的发展，提升自己在这个领域中的竞争力。"

Table of Contents

[ v ]

Developing YARN applications 162

Writing YARN clients 162

Writing the Application Master entity 169

Monitoring YARN 175

Job scheduling in YARN 181

CapacityScheduler 181

FairScheduler 184

YARN commands 188

User commands 188

Administration commands 189

Summary 190

Chapter 7: Storm on YARN – Low Latency

Processing in Hadoop 191

Batch processing versus streaming 192

Apache Storm 193

Architecture of an Apache Storm cluster 194

Computation and data modeling in Apache Storm 196

Use cases for Apache Storm 197

Developing with Apache Storm 198

Apache Storm 0.9.1 206

Storm on YARN 207

Installing Apache Storm-on-YARN 207

Prerequisites 207

Installation procedure 208

Summary 218

Chapter 8: Hadoop on the Cloud 219

Cloud computing characteristics 219

Hadoop on the cloud 221

Amazon Elastic MapReduce (EMR) 222

Provisioning a Hadoop cluster on EMR 223

Summary 237

Chapter 9: HDFS Replacements 239

HDFS – advantages and drawbacks 240

Amazon AWS S3 240

Hadoop support for S3 241

Implementing a lesystem in Hadoop 243

Implementing an S3 native lesystem in Hadoop 244

Summary 256

Preface

We are in an age where data is the primary driver in decision-making. With storage

costs declining, network speeds increasing, and everything around us becoming

digital, we do not hesitate a bit to download, store, or share data with others around

us. About 20 years back, a camera was a device used to capture pictures on lm.

Every photograph had to be captured almost perfectly. The storage of lm negatives

was done carefully lest they get damaged. There was a higher cost associated with

taking prints of these photographs. The time taken between a picture click and to

view it was almost a day. This meant that less data was being captured as these

factors presented a cliff for people from recording each and every moment of their

life, unless it was very signicant.

However, with cameras becoming digital, this has changed. We do not hesitate to

click a photograph of almost anything anytime. We do not worry about storage

as our externals disks of a terabyte capacity always provide a reliable backup. We

seldom take our cameras anywhere as we have mobile devices that we can use to

take photographs. We have applications such as Instagram that can be used to add

effects to our pictures and share them. We gather opinions and information about the

pictures, and we click and base some of our decisions on them. We capture almost

every moment, of great signicance or not, and push it into our memory books. The

era of big data has arrived!

This era of Big Data has similar changes in businesses as well. Almost everything in a

business is logged. Every action taken by a user on the page of an e-commerce page is

recorded to improve quality of service and every item bought by the user are recorded

to cross-sell or up-sell other items. Businesses want to understand the DNA of their

customers and try to infer it by pinching out every possible data they can get about

these customers. Businesses are not worried about the format of the data. They are

ready to accept speech, images, natural language text, or structured data. These data

points are used to drive business decisions and personalize experiences for the user.

The more data, the higher the degree of personalization and better the experience for

the user.

剩余373页未读，继续阅读

DoomLord

粉丝: 114
资源: 1318

掌握下一代Hadoop数据处理平台：进阶指南

使用Hadoop解决实际问题：实战指南

探索Hadoop2：存储与计算的革命

精通Flask Web开发（第二版）

Packt.Mastering.Apache.Spark

Packt.Mastering.Internet.of.Things.2018

英文原版 packt Mastering Hadoop

Mastering.Apache.Cassandra.2nd.Edition.1784392618

Mastering Apache Spark

Mastering Apache Solr 7.x An expert guide to advancing, optimizing, 无水印转化版pdf

Mastering Elasticsearch, Second Edition

最新资源