HadoopSuccinctly - CSDN文库

Hadoop

需积分: 10 81 浏览量更新于2023-03-16 评论收藏 2.42MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

资源详情

资源评论

资源推荐

 
4 
Table of Contents 
About the Author ..................................................................................................................... 7 
Chapter 1  Introducing Hadoop ..............................................................................................10 
What is Hadoop? ..................................................................................................................10 
Do I have Big Data? .............................................................................................................10 
How Hadoop works ..............................................................................................................12 
Summary ..............................................................................................................................14 
Chapter 2  Getting Started with Hadoop ................................................................................16 
Running Hadoop in a Docker container ................................................................................16 
Verifying your installation ......................................................................................................17 
The structure of a MapReduce program ...............................................................................18 
The word count mapper ........................................................................................................20 
The word count reducer ........................................................................................................23 
The word count driver ...........................................................................................................24 
Running the MapReduce program ........................................................................................26 
Summary ..............................................................................................................................27 
Chapter 3  HDFS—The Hadoop Distributed File System ......................................................28 
Introducing HDFS .................................................................................................................28 
Architecture of HDFS............................................................................................................28 
Accessing HDFS from the command line .............................................................................29 
Write once, read many .........................................................................................................32 
File splits and replication ......................................................................................................33 
HDFS Client and the web UI .................................................................................................35 
Summary ..............................................................................................................................35 
Chapter 4  YARN—Yet Another Resource Negotiator ..........................................................37 

 
5 
Introducing YARN .................................................................................................................37 
The structure of a YARN job .................................................................................................37 
How MapReduce works in YARN .........................................................................................39 
Scheduler implementations ..................................................................................................39 
What is a resource? .............................................................................................................42 
Monitoring jobs in YARN ......................................................................................................42 
Summary ..............................................................................................................................44 
Chapter 5  Hadoop Streaming ................................................................................................45 
Introducing Hadoop Streaming .............................................................................................45 
Streaming jobs in Hadoop ....................................................................................................45 
The streaming interface ........................................................................................................46 
Streaming word count in Python ...........................................................................................47 
Streaming word count in .NET ..............................................................................................51 
Interoperability with Hadoop Streaming ................................................................................55 
Managing dependencies in Hadoop Streaming ....................................................................55 
Performance impact of Hadoop Streaming ...........................................................................56 
Summary ..............................................................................................................................57 
Chapter 6  Inside the Cluster ..................................................................................................58 
Introduction ..........................................................................................................................58 
Sizing Hadoop—minimal cluster ...........................................................................................58 
Sizing Hadoop—the small cluster .........................................................................................59 
Sizing Hadoop – large clusters .............................................................................................61 
Reliability in HDFS ...............................................................................................................61 
Reliability in YARN ...............................................................................................................63 
Cluster configuration ............................................................................................................63 
core-site.xml .........................................................................................................................64 