Hadoop运维实战：Eric Sammer著

需积分: 10 40 浏览量更新于2024-07-26 收藏 8.02MB PDF 举报

"Hadoop.Operations 是一本由 Eric Sammer 编写的关于 Hadoop 操作的电子书，适合学习和理解 Hadoop 的运维知识。这本书涵盖了 Hadoop 的核心组件，包括 MapReduce，以及与之相关的实际操作和管理技巧。" 《Hadoop Operations》一书深入探讨了 Hadoop 生态系统中的关键组件和最佳实践，为那些负责部署、管理和优化 Hadoop 集群的专业人士提供了宝贵的指导。作者 Eric Sammer 在书中分享了他的经验，帮助读者理解如何在生产环境中有效地运行 Hadoop。 1. **Hadoop 简介**：Hadoop 是一个开源的分布式计算框架，基于 Google 的 MapReduce 和 GFS（Google 文件系统）概念设计。它允许在大规模数据集上进行并行处理，具有高容错性和可扩展性，是大数据处理的核心工具之一。 2. **MapReduce**：MapReduce 是 Hadoop 中用于处理和生成大数据集的编程模型。它将大型任务分解为小任务（map阶段），并在集群中的多台机器上并行处理，然后将结果整合（reduce阶段）。书中会详细解释 MapReduce 的工作原理、编程模型以及优化策略。 3. **Hadoop 部署**：书中涵盖了 Hadoop 集群的规划、安装和配置，包括硬件选择、网络架构、数据节点和名称节点的设置等。此外，还会讨论单机模式、伪分布式模式和完全分布式模式的区别和应用。 4. **Hadoop 管理**：运维方面，书中有对监控、故障排查、性能调优、数据备份和恢复等主题的深入讨论。这包括使用如 Ganglia、Nagios 这样的监控工具，以及如何应对 HDFS 和 MapReduce 的常见问题。 5. **安全性与权限**：在大数据环境中，安全性是至关重要的。书中的这部分将介绍如何实施 Kerberos 认证、Hadoop 安全模式以及如何管理用户和组权限。 6. **Hadoop 扩展性与容错性**：Hadoop 设计的目标就是高可用性和容错性。书中将讲解如何通过增加节点来扩展集群，以及如何利用 HA（高可用性）功能确保服务不中断。 7. **Hadoop 相关工具**：除了 MapReduce，Hadoop 生态系统还包括许多其他工具，如 HBase（分布式数据库）、Hive（数据仓库工具）、Pig（数据分析平台）等。这些工具如何与 MapReduce 结合使用，也会在书中有所涉及。 8. **案例研究**：书中可能包含真实世界的案例研究，展示 Hadoop 在不同行业的应用，帮助读者了解如何将理论知识应用于实际项目。 9. **最佳实践**：最后，作者会分享一些最佳实践，包括集群维护、代码优化、数据处理策略等，以帮助读者避免常见陷阱，提升 Hadoop 集群的性能和稳定性。《Hadoop Operations》是一本全面的指南，涵盖了从基础到高级的 Hadoop 知识，对于任何希望深入了解和掌握 Hadoop 运维的人来说都是不可或缺的资源。

and generally improve in quality with more of it. Knowing more about a problem space

generally leads to better decisions (or algorithm efficacy), which in turn leads to happier

users, more money, reduced fraud, healthier people, safer conditions, or whatever the

desired result might be.

Apache Hadoop is a platform that provides pragmatic, cost-effective, scalable infra-

structure for building many of the types of applications described earlier. Made up of

a distributed filesystem called the Hadoop Distributed Filesystem (HDFS) and a com-

putation layer that implements a processing paradigm called MapReduce, Hadoop is

an open source, batch data processing system for enormous amounts of data. We live

in a flawed world, and Hadoop is designed to survive in it by not only tolerating hard-

ware and software failures, but also treating them as first-class conditions that happen

regularly. Hadoop uses a cluster of plain old commodity servers with no specialized

hardware or network infrastructure to form a single, logical, storage and compute plat-

form, or cluster, that can be shared by multiple individuals or groups. Computation in

Hadoop MapReduce is performed in parallel, automatically, with a simple abstraction

for developers that obviates complex synchronization and network programming. Un-

like many other distributed data processing systems, Hadoop runs the user-provided

processing logic on the machine where the data lives rather than dragging the data

across the network; a huge win for performance.

For those interested in the history, Hadoop was modeled after two papers produced

by Google, one of the many companies to have these kinds of data-intensive processing

problems. The first, presented in 2003, describes a pragmatic, scalable, distributed

filesystem optimized for storing enormous datasets, called the Google Filesystem, or

GFS. In addition to simple storage, GFS was built to support large-scale, data-intensive,

distributed processing applications. The following year, another paper, titled "Map-

Reduce: Simplified Data Processing on Large Clusters," was presented, defining a pro-

gramming model and accompanying framework that provided automatic paralleliza-

tion, fault tolerance, and the scale to process hundreds of terabytes of data in a single

job over thousands of machines. When paired, these two systems could be used to build

large data processing clusters on relatively inexpensive, commodity machines. These

papers directly inspired the development of HDFS and Hadoop MapReduce, respec-

tively.

Interest and investment in Hadoop has led to an entire ecosystem of related software

both open source and commercial. Within the Apache Software Foundation alone,

projects that explicitly make use of, or integrate with, Hadoop are springing up regu-

larly. Some of these projects make authoring MapReduce jobs easier and more acces-

sible, while others focus on getting data in and out of HDFS, simplify operations, enable

deployment in cloud environments, and so on. Here is a sampling of the more popular

projects with which you should familiarize yourself:

Apache Hive

Hive creates a relational database−style abstraction that allows developers to write

a dialect of SQL, which in turn is executed as one or more MapReduce jobs on the

2 | Chapter 1: Introduction

cluster. Developers, analysts, and existing third-party packages already know and

speak SQL (Hive’s dialect of SQL is called HiveQL and implements only a subset

of any of the common standards). Hive takes advantage of this and provides a quick

way to reduce the learning curve to adopting Hadoop and writing MapReduce jobs.

For this reason, Hive is by far one of the most popular Hadoop ecosystem projects.

Hive works by defining a table-like schema over an existing set of files in HDFS

and handling the gory details of extracting records from those files when a query

is run. The data on disk is never actually changed, just parsed at query time. HiveQL

statements are interpreted and an execution plan of prebuilt map and reduce

classes is assembled to perform the MapReduce equivalent of the SQL statement.

Apache Pig

Like Hive, Apache Pig was created to simplify the authoring of MapReduce jobs,

obviating the need to write Java code. Instead, users write data processing jobs in

a high-level scripting language from which Pig builds an execution plan and exe-

cutes a series of MapReduce jobs to do the heavy lifting. In cases where Pig doesn’t

support a necessary function, developers can extend its set of built-in operations

by writing user-defined functions in Java (Hive supports similar functionality as

well). If you know Perl, Python, Ruby, JavaScript, or even shell script, you can learn

Pig’s syntax in the morning and be running MapReduce jobs by lunchtime.

Apache Sqoop

Not only does Hadoop not want to replace your database, it wants to be friends

with it. Exchanging data with relational databases is one of the most popular in-

tegration points with Apache Hadoop. Sqoop, short for “SQL to Hadoop,” per-

forms bidirectional data transfer between Hadoop and almost any database with

a JDBC driver. Using MapReduce, Sqoop performs these operations in parallel

with no need to write code.

For even greater performance, Sqoop supports database-specific plug-ins that use

native features of the RDBMS rather than incurring the overhead of JDBC. Many

of these connectors are open source, while others are free or available from com-

mercial vendors at a cost. Today, Sqoop includes native connectors (called direct

support) for MySQL and PostgreSQL. Free connectors exist for Teradata, Netezza,

SQL Server, and Oracle (from Quest Software), and are available for download

from their respective company websites.

Apache Flume

Apache Flume is a streaming data collection and aggregation system designed to

transport massive volumes of data into systems such as Hadoop. It supports native

connectivity and support for writing directly to HDFS, and simplifies reliable,

streaming data delivery from a variety of sources including RPC services, log4j

appenders, syslog, and even the output from OS commands. Data can be routed,

load-balanced, replicated to multiple destinations, and aggregated from thousands

of hosts by a tier of agents.

Introduction | 3

Apache Oozie

It’s not uncommon for large production clusters to run many coordinated Map-

Reduce jobs in a workfow. Apache Oozie is a workflow engine and scheduler built

specifically for large-scale job orchestration on a Hadoop cluster. Workflows can

be triggered by time or events such as data arriving in a directory, and job failure

handling logic can be implemented so that policies are adhered to. Oozie presents

a REST service for programmatic management of workflows and status retrieval.

Apache Whirr

Apache Whirr was developed to simplify the creation and deployment of ephem-

eral clusters in cloud environments such as Amazon’s AWS. Run as a command-

line tool either locally or within the cloud, Whirr can spin up instances, deploy

Hadoop, configure the software, and tear it down on demand. Under the hood,

Whirr uses the powerful jclouds library so that it is cloud provider−neutral. The

developers have put in the work to make Whirr support both Amazon EC2 and

Rackspace Cloud. In addition to Hadoop, Whirr understands how to provision

Apache Cassandra, Apache ZooKeeper, Apache HBase, ElasticSearch, Voldemort,

and Apache Hama.

Apache HBase

Apache HBase is a low-latency, distributed (nonrelational) database built on top

of HDFS. Modeled after Google’s Bigtable, HBase presents a flexible data model

with scale-out properties and a very simple API. Data in HBase is stored in a semi-

columnar format partitioned by rows into regions. It’s not uncommon for a single

table in HBase to be well into the hundreds of terabytes or in some cases petabytes.

Over the past few years, HBase has gained a massive following based on some very

public deployments such as Facebook’s Messages platform. Today, HBase is used

to serve huge amounts of data to real-time systems in major production deploy-

ments.

Apache ZooKeeper

A true workhorse, Apache ZooKeeper is a distributed, consensus-based coordina-

tion system used to support distributed applications. Distributed applications that

require leader election, locking, group membership, service location, and config-

uration services can use ZooKeeper rather than reimplement the complex coordi-

nation and error handling that comes with these functions. In fact, many projects

within the Hadoop ecosystem use ZooKeeper for exactly this purpose (most no-

tably, HBase).

Apache HCatalog

A relatively new entry, Apache HCatalog is a service that provides shared schema

and data access abstraction services to applications with the ecosystem. The

long-term goal of HCatalog is to enable interoperability between tools such as

Apache Hive and Pig so that they can share dataset metadata information.

The Hadoop ecosystem is exploding into the commercial world as well. Vendors such

as Oracle, SAS, MicroStrategy, Tableau, Informatica, Microsoft, Pentaho, Talend, HP,

4 | Chapter 1: Introduction

Dell, and dozens of others have all developed integration or support for Hadoop within

one or more of their products. Hadoop is fast becoming (or, as an increasingly growing

group would believe, already has become) the de facto standard for truly large-scale

data processing in the data center.

If you’re reading this book, you may be a developer with some exposure to Hadoop

looking to learn more about managing the system in a production environment. Alter-

natively, it could be that you’re an application or system administrator tasked with

owning the current or planned production cluster. Those in the latter camp may be

rolling their eyes at the prospect of dealing with yet another system. That’s fair, and we

won’t spend a ton of time talking about writing applications, APIs, and other pesky

code problems. There are other fantastic books on those topics, especially Hadoop: The

Definitive Guide by Tom White (O’Reilly). Administrators do, however, play an abso-

lutely critical role in planning, installing, configuring, maintaining, and monitoring

Hadoop clusters. Hadoop is a comparatively low-level system, leaning heavily on the

host operating system for many features, and it works best when developers and ad-

ministrators collaborate regularly. What you do impacts how things work.

It’s an extremely exciting time to get into Apache Hadoop. The so-called big data space

is all the rage, sure, but more importantly, Hadoop is growing and changing at a stag-

gering rate. Each new version—and there have been a few big ones in the past year or

two—brings another truckload of features for both developers and administrators

alike. You could say that Hadoop is experiencing software puberty; thanks to its rapid

growth and adoption, it’s also a little awkward at times. You’ll find, throughout this

book, that there are significant changes between even minor versions. It’s a lot to keep

up with, admittedly, but don’t let it overwhelm you. Where necessary, the differences

are called out, and a section in Chapter 4 is devoted to walking you through the most

commonly encountered versions.

This book is intended to be a pragmatic guide to running Hadoop in production. Those

who have some familiarity with Hadoop may already know alternative methods for

installation or have differing thoughts on how to properly tune the number of map slots

based on CPU utilization.

That’s expected and more than fine. The goal is not to

enumerate all possible scenarios, but rather to call out what works, as demonstrated

in critical deployments.

Chapters 2 and 3 provide the necessary background, describing what HDFS and Map-

Reduce are, why they exist, and at a high level, how they work. Chapter 4 walks you

through the process of planning for an Hadoop deployment including hardware selec-

tion, basic resource planning, operating system selection and configuration, Hadoop

distribution and version selection, and network concerns for Hadoop clusters. If you

are looking for the meat and potatoes, Chapter 5 is where it’s at, with configuration

and setup information, including a listing of the most critical properties, organized by

2. We also briefly cover the flux capacitor and discuss the burn rate of energon cubes during combat.

Introduction | 5

剩余296页未读，继续阅读

sdautianzhiyi

粉丝: 8
资源: 48

Hadoop运维实战：Eric Sammer著

NetBackup10_AdminGuide_Hadoop.pdf

NetBackup101_AdminGuide_Hadoop.pdf

hadoop_operations

Hadoop Operations (2012.9)

spark-1.3.1-bin-hadoop2.6.tgz

Apache Hadoop 3.x state of the union and upgrade guidance

Hadoop Operations 实用指南

Hadoop Operations：权威指南

Hadoop Operations：云计算实战指南

Hadoop Operations：探索2.2新特性

最新资源