Apache Flume：日志收集利器，无缝对接Hadoop

flume

需积分: 9 31 浏览量更新于2023-06-04 收藏 3.69MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

资源详情

资源推荐

Preface

[ 5 ]

Piracy

Piracy of copyright material on the Internet is an ongoing problem across all media.

At Packt, we take the protection of our copyright and licenses very seriously. If you

come across any illegal copies of our works, in any form, on the Internet, please

provide us with the location address or website name immediately so that we can

pursue a remedy.

Please contact us at

pirated material.

We appreciate your help in protecting our authors, and our ability to bring you

valuable content.

Questions

You can contact us at questions@packtpub.com if you are having a problem with

any aspect of the book, and we will do our best to address it.

Overview and Architecture

[ 8 ]

Flume 0.9

Flume was rst introduced in Cloudera's CDH3 Distribution in 2011. It consisted

of a federation of worker daemons (agents) congured from a centralized master

(or masters) via Zookeeper (a federated conguration and coordination system).

From the master you could check agent status in a Web UI, as well as push out

conguration centrally from the UI or via a command line shell (both really

communicating via Zookeeper to the worker agents).

Data could be sent in one of the three modes, namely, best effort (BE), disk failover

(DFO), and end-to-end (E2E). The masters were used for the end-to-end (E2E) mode

acknowledgements and multi-master conguration never really matured so usually

you had only one master making it a central point of failure for E2E data ows.

Best effort is just what it sounds like—the agent would try and send the data, but if

it couldn't, the data would be discarded. This mode is good for things like metrics

where gaps can easily be tolerated, as new data is just a second away. Disk failover

mode stores undeliverable data to the local disk (or sometimes a local database)

and keeps retrying until the data can be delivered to the next recipient in your data

ow. This is handy for those planned (or unplanned) outages as long as you have

sufcient local disk space to buffer the load.

In June of 2011, Cloudera moved control of the Flume project to the Apache

foundation. It came out of incubator status a year later in 2012. During that

incubation year, work had already begun to refactor Flume under the Star Trek

Themed tag, Flume-NG (Flume the Next Generation).

Flume 1.X (Flume-NG)

There were many reasons to why Flume was refactored. If you are interested in

the details you can read about it at https://issues.apache.org/jira/browse/

FLUME-728

. What started as a refactoring branch eventually became the main line

of development as Flume 1.X.

The most obvious change in Flume 1.X is that the centralized conguration master/

masters and Zookeeper are gone. The conguration in Flume 0.9 was overly verbose

and mistakes were easy to make. Furthermore, centralized conguration was really

outside the scope of Flume's goals. Centralized conguration was replaced with

a simple on-disk conguration le (although the conguration provider is pluggable

so that it can be replaced). These conguration les are easily distributed using tools

such as cf-engine, chef, and puppet. If you are using a Cloudera Distribution, take

a look at Cloudera Manager to manage your congurations—their licensing was

recently changed to lift the node limit so it may be an attractive option for you.

Be sure you don't manage these congurations manually or you'll be editing those

les manually forever.

Chapter 1

[ 9 ]

Another major difference in Flume 1.X is that the reading of input data and the

writing of output data are now handled by different worker threads (called Runners).

In Flume 0.9, the input thread also did the writing to the output (except for failover

retries). If the output writer was slow (rather than just failing outright), it would block

Flume's ability to ingest data. This new asynchronous design leaves the input thread

blissfully unaware of any downstream problem.

The version of Flume covered in this book is 1.3.1 (current at the time of this

book's writing).

The problem with HDFS and streaming

data/logs

HDFS isn't a real lesystem, at least not in the traditional sense, and many of the

things we take for granted with normal lesystems don't apply here, for example

being able to mount it. This makes getting your streaming data into Hadoop a little

more complicated.

In a regular Portable Operating System Interface (POSIX) style lesystem, if you

open a le and write data, it still exists on disk before the le is closed. That is, if

another program opens the same le and starts reading, it will get the data already

ushed by the writer to disk. Furthermore, if that writing process is interrupted,

any portion that made it to disk is usable (it may be incomplete, but it exists).

In HDFS the le exists only as a directory entry, it shows as having zero length until

the le is closed. This means if data is written to a le for an extended period without

closing it, a network disconnect with the client will leave you with nothing but an

empty le for all your efforts. This may lead you to the conclusion that it would be

wise to write small les so you can close them as soon as possible.

The problem is Hadoop doesn't like lots of tiny les. Since the HDFS metadata is

kept in memory on the NameNode, the more les you create, the more RAM you'll

need to use. From a MapReduce prospective, tiny les lead to poor efciency.

Usually, each mapper is assigned a single block of a le as input (unless you have

used certain compression codecs). If you have lots of tiny les, the cost of starting

the worker processes can be disproportionally high compared to the data it is

processing. This kind of block fragmentation also results in more mapper tasks

increasing the overall job run times.

剩余107页未读，继续阅读

terencecpp

粉丝: 1
资源: 33

会员权益专享

Apache Flume：日志收集利器，无缝对接Hadoop

Apache Flume Distributed Log Collection for Hadoop 高清完整中文版PDF下载

Apache Flume Distributed Log Collection for Hadoop（中文版）

Apache Flume, Distributed Log Collection for Hadoop（第二版）

Apache Flume所需依赖添加到pom.xml文件中

Apache Flume需要的依赖

apache-flume-1.9.0-bin.tar.gz下载

在本地idea上创建一个Java项目，添加Apache Flume的依赖。

bash: /usr/local/flume/log.00: Permission denied

windows下flume采集log日志

log4j:WARN No appenders could be found for logger (org.apache.flume.util.SSLUtil).

apache flume原理

log4j:WARN No appenders could be found for logger (org.apache.flume.node.PollingPropertiesF

使用PyCharm编程实现SparkCore的WordCount单词统计基础，并保存在HDFS或本地文件中 注意：新建文件为words.txt，文件路径在/export/server/data下面，内容如下 Spark Flink flume hadoop Flink spark flume hadoop

hadoop的flume

./bin/flume-ng version没有反应

将 flume 文件先拷贝到桌面上，在移动到/usr/local 下，在/usr/local 解压 tar -zxvf apache-flume-1.7.0-bin.tar.gz mv apache-flume-1.7.0-bin flume #改名

flume log4j:WARN No appenders could be found for logger (org.apache.flume.node.PollingPropertiesF

错误: 找不到或无法加载主类 org.apache.flume.tools.GetJavaProperty 错误: 找不到或无法加载主类 org.apache.hadoop.hbase.util.GetJavaProperty

Apache Flume- Distributed Log Collection for Hadoop(PACKT,2013)

Apache Flume Distributed Log Collection for Hadoop(PACKT,2ed,2015)

会员权益专享

最新资源

使用PyCharm编程实现SparkCore的WordCount单词统计基础，并保存在HDFS或本地文件中注意：新建文件为words.txt，文件路径在/export/server/data下面，内容如下 Spark Flink flume hadoop Flink spark flume hadoop