使用Ganglia监控Hadoop

需积分: 9 143 浏览量更新于2024-07-25 收藏 4.27MB PDF 举报

"监控Hadoop的Ganglia工具" Ganglia是用于监控分布式系统，包括Hadoop在内的一个开源监控框架。这本书《Monitoring with Ganglia》由Matt Massie、Bernard Li、Brad Nicholes和Vladimir Vuksan共同撰写，详细介绍了如何利用Ganglia来监控复杂的IT环境。Ganglia提供了实时的性能数据收集和可视化，帮助管理员识别和解决问题，确保系统的高效运行。 Ganglia的核心功能包括： 1. **分布式监控**：Ganglia能够跨多台主机收集性能数据，支持大规模的集群环境，包括Hadoop集群。 2. **实时数据**：它提供持续的、低延迟的数据收集，使管理员能够实时了解系统的健康状况和性能指标。 3. **自定义指标**：用户可以根据需要定义自己的监控指标，监控特定的服务或应用性能。 4. **可视化界面**：Ganglia提供直观的Web界面，显示图形化的性能报告，包括CPU利用率、内存使用、网络流量等。 5. **可扩展性**：Ganglia设计时考虑了可扩展性，可以轻松添加或删除被监控节点，适应动态变化的环境。 6. **模块化结构**：Ganglia由多个组件组成，如gmetad（元数据聚合服务器）、gmond（数据收集代理）和gweb（Web接口），这些组件可以通过配置进行组合以满足不同需求。 7. **性能优化**：Ganglia使用高效的数据传输协议（如UDP）和压缩算法，降低网络负载，同时保证数据的准确性。在使用Ganglia监控Hadoop时，你可以关注以下关键指标： - **MapReduce作业状态**：包括作业进度、失败任务、任务执行时间等。 - **NameNode状态**：如文件系统命名空间的健康状况、块的复制率等。 - **DataNode状态**：磁盘使用情况、数据块的读写速率、内存占用等。 - **JobTracker和TaskTracker**：它们的CPU和内存使用，以及任务调度情况。通过Ganglia，Hadoop管理员可以及时发现和解决性能瓶颈，优化集群配置，提高整体效率。同时，Ganglia还支持与其他监控工具集成，如Nagios和Zabbix，实现更全面的监控解决方案。这本书《Monitoring with Ganglia》详细介绍了Ganglia的安装、配置、维护和故障排查，是理解并有效利用Ganglia监控Hadoop和其他分布式系统的宝贵资源。

Constant width

Used for program listings, as well as within paragraphs to refer to program elements

such as variable or function names, databases, data types, environment variables,

statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values deter-

mined by context.

This icon signifies a tip, suggestion, or general note.

This icon indicates a warning or caution.

Using Code Examples

This book is here to help you get your job done. In general, you may use the code in

this book in your programs and documentation. You do not need to contact us for

permission unless you’re reproducing a significant portion of the code. For example,

writing a program that uses several chunks of code from this book does not require

permission. Selling or distributing a CD-ROM of examples from O’Reilly books does

require permission. Answering a question by citing this book and quoting example

code does not require permission. Incorporating a significant amount of example code

from this book into your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title,

author, publisher, and ISBN. For example: “Monitoring with Ganglia by Matt Massie,

Massie, Bernard Li, Brad Nicholes, Vladimir Vuksan, 978-1-449-32970-9.”

If you feel your use of code examples falls outside fair use or the permission given above,

feel free to contact us at permissions@oreilly.com.

Safari® Books Online

Safari Books Online (www.safaribooksonline.com) is an

on-demand digital

library that delivers expert content in both book and video form from the

world’s leading authors in technology and business.

xiv | Preface

CHAPTER 1

Introducing Ganglia

Dave Josephsen

you’re reading this, odds are you have a problem to solve. I won’t presume to guess

the particulars, but I’m willing to bet that the authors of this book have shared your

pain at one time or another, so if you’re in need of a monitoring and metrics collection

engine, you’ve come to the right place. We created Ganglia for the same reason you’ve

picked up this book: we had a problem to solve.

If you’ve looked at other monitoring tools, or have already implemented a few, you’ll

find that Ganglia is as powerful as it is conceptually and operationally different from

any monitoring system you’re likely to have previously encountered. It runs on every

popular OS out there, scales easily to very large networks, and is resilient by design to

node failures. In the real world, Ganglia routinely provides near real-time monitoring

and performance metrics data for computer networks that are simply too large for more

traditional monitoring systems to handle, and it integrates seamlessly with any tradi-

tional monitoring systems you may happen to be using.

In this chapter, we’d like to introduce you to Ganglia and help you evaluate whether

it’s a good fit for your environment. Because Ganglia is a product of the labor of systems

guys—like you—who were trying to solve a problem, our introduction begins with a

description of the environment in which Ganglia was born and the problem it was

intended to solve.

It’s a Problem of Scale

Say you have a lot of machines. I’m not talking a few hundred, I mean metric oodles of

servers, stacked floor to ceiling as far as the eye can see. Servers so numerous that they

put to shame swarms of locusts, outnumber the snowflakes in Siberia, and must be

expressed in scientific notation, or as some multiple of Avogadro’s number.

Okay, maybe not quite that numerous, but the point is, if you had lots of machines,

how would you go about gathering a metric—the CPU utilization, say—from every

host every 10 seconds? Assuming 20,000 hosts, for example, your monitoring system

would need to poll 2,000 hosts per second to achieve a 10-second resolution for that

singular metric. It would also need to store, graph, and present that data quickly and

efficiently. This is the problem domain for which Ganglia was designed; to monitor

and collect massive quantities of system metrics in near real time for Large installations.

Large. With a capital L.

Large installations are interesting because they force us to reinvent or at least reevaluate

every problem we thought we’d already solved as systems administrators. The prospect

of firing up rsync or kludging together some Perl is altogether different when 20,000

hosts are involved. As the machines become more numerous, we’re more likely to care

about the efficiency of the polling protocol, we’re more likely to encounter exceptions,

and we’re less likely to interact directly with every machine. That’s not even mentioning

the quadratic curve towards infinity that describes the odds of some subset of our hosts

going offline as the total number grows.

I don’t mean to imply that Ganglia can’t be used in smaller networks—swarms of

locusts would laugh at my own puny corporate network and I couldn’t live without

Ganglia—but it’s important to understand the design characteristics from which Gan-

glia was derived, because as I mentioned, Ganglia operates quite differently from other

monitoring systems because of them. The most influential consideration shaping Gan-

glia’s design is certainly the problem of scale.

Hosts ARE the Monitoring System

The problem of scale also changes how we think about systems management, some-

times in surprising or counterintuitive ways. For example, an admin over 20,000

systems is far more likely to be running a configuration management engine such as

Puppet/Chef or CFEngine and will therefore have fewer qualms about host-centric

configuration. The large installation administrator knows that he can make configu-

ration changes to all of the hosts centrally. It’s no big deal. Smaller installations instead

tend to favor tools that minimize the necessity to configure individual hosts.

Large installation admin are rarely concerned about individual node failures. Designs

that incorporate single points of failure are generally to be avoided in large application

frameworks where it can be safely assumed, given the sheer amount of hardware in-

volved, that some percentage of nodes are always going to be on the fritz. Smaller

installations tend to favor monitoring tools that strictly define individual hosts centrally

and alert on individual host failures. This sort of behavior quickly becomes unwieldy

and annoying in larger networks.

If you think about it, the monitoring systems we’re used to dealing with all work the

way they do because of this “little network” mind set. This tendency to centralize and

strictly define the configuration begets a central daemon that sits somewhere on the

network and polls every host every so often for status. These systems are easy to use in

small environments: just install the (usually bloated) agent on every system and

2 | Chapter 1: Introducing Ganglia

剩余253页未读，继续阅读

deamszhang

粉丝: 0
资源: 1

使用Ganglia监控Hadoop

Hadoop性能监控：JMX与Ganglia结合实践

Hadoop管理：掌握HDFS操作与Ganglia监控

一键部署：Rancher容器化Ganglia监控Hadoop集群

ganglia

Linux 5 下安装Ganglia 3.0.3

Big Data Made Easy - A Working Guide To The Complete Hadoop Toolset

云计算驱动：快速搭建Apache Hadoop集群指南

阿里搜索计算平台：Hadoop与YARN的演进

Hadoop数据节点磁盘管理与IO性能优化

Hadoop NameNode日志分析：系统健康状况深度挖掘

最新资源