Jay Kreps著《I Heart Logs》：事件数据与流处理指南

需积分: 9 32 浏览量更新于2024-07-17 收藏 3.9MB PDF 举报

《I Heart Logs》是由Jay Kreps撰写的一本专著，全称为"I Heart Logs: Event Data, Stream Processing, and the Quest for Real-Time Insights"。这本书在2015年首次出版，享有版权，并由O'Reilly Media公司发行，地址位于美国加利福尼亚州塞巴斯托波尔。本书的目标读者群体是那些对事件数据、流处理以及实时洞察感兴趣的IT专业人士，特别适合教育、商业或销售推广用途。这本书的核心内容围绕着日志分析在现代IT环境中的重要性，强调了日志作为一种关键的数据源，如何帮助我们理解和优化系统的运行情况。作者Jay Kreps以其丰富的经验，深入浅出地讲解了以下知识点： 1. **事件数据**：书中详细阐述了事件数据的概念，包括它与结构化数据的区别，以及在分布式系统和微服务架构中如何收集和处理这些数据。 2. **流处理**：作者讨论了实时流处理技术，如Apache Kafka、Apache Flink和Apache Spark Streaming等，以及它们如何实现实时数据处理和分析，以支持业务决策和故障检测。 3. **日志监控与分析**：书中介绍了一系列实用的日志分析工具和技术，如ELK Stack（Elasticsearch、Logstash和Kibana）以及Logstash插件，帮助读者理解如何通过日志来发现模式、异常和性能瓶颈。 4. **实时洞察**：重点讲述了如何利用实时数据流来生成即时的业务洞察，这对于故障预防、性能优化和持续改进至关重要。 5. **实践案例**：书中包含了许多实际案例研究，展示了在不同行业和场景中如何有效地应用日志分析和流处理技术，以便读者能更好地将理论知识转化为实际操作。 6. **最佳实践和设计原则**：作者分享了他在IT领域的宝贵经验，提出了关于日志架构、监控策略以及数据模型设计的关键原则，帮助读者构建健壮且可扩展的实时数据处理系统。《I Heart Logs》不仅是一本理论著作，还提供了丰富的参考资料和链接，方便读者在线获取更多信息。此外，书中的修订历史记录了作者在2014年9月22日首次发布以来的更新情况，确保了内容的时效性和准确性。总体而言，《I Heart Logs》是一本深入解析日志管理和实时数据分析的实战指南，对于任何寻求提升IT系统可见性、理解和响应能力的专业人士来说，都是一份宝贵的资源。

What Is a Log?

When most people think about logs they probably think about something that looks like

Figure 1-1.

Every programmer is familiar with this kind of log — a series of loosely structured

requests, errors, or other messages in a sequence of rotating text files.

This type of log is a degenerative form of the log concept I am going to describe. The

biggest difference is that this type of application log is mostly meant for humans to read,

whereas the logs I’ll be describing are also for programmatic access.

Actually, if you think about it, the idea of humans reading through logs on individual

machines is something of an anachronism. This approach quickly becomes unmanageable

when many services and servers are involved. The purpose of logs quickly becomes an

input to queries and graphs in order to understand behavior across many machines,

something that English text in files is not nearly as appropriate for as the kind of

structured log I’ll be talking about.

Figure 1-1. An excerpt from an Apache log

The log I’ll be discussing is a little more general and closer to what in the database or

systems world might be called a commit log or journal. It is an append-only sequence of

records ordered by time, as in Figure 1-2.

Logs in Databases

I don’t know where the log concept originated — it is probably one of those things like

binary search that is too simple for the inventor to realize it was an invention. It is present

as early as IBM’s System R. The usage in databases has to do with keeping in sync a

variety of data structures and indexes in the presence of crashes. To make this atomic and

durable, a database uses a log to write out information about the records it will be

modifying before applying the changes to all the various data structures that it maintains.

The log is the record of what happened, and each table or index is a projection of this

history into some useful data structure or index. Since the log is immediately persisted, it

is used as the authoritative source in restoring all other persistent structures in the event of

a crash.

Over time, the usage of the log grew from an implementation detail of the ACID database

properties (atomicity, consistency, isolation, and durability) to a method for replicating

data between databases. It turns out that the sequence of changes that happened on the

database is exactly what is needed to keep a remote replica database in sync. Oracle,

MySQL, PostgreSQL, and MongoDB include log shipping protocols to transmit portions

of a log to replica databases that act as slaves. The slaves can then apply the changes

recorded in the log to their own local data structures to stay in sync with the master.

Oracle has productized the log as a general data subscription mechanism for non-Oracle

data subscribers with their XStreams and GoldenGate products, and similar facilities exist

in MySQL and PostgreSQL.

In fact, the use of logs in much of the rest of this book will be variations on the two uses in

database internals:

1. The log is used as a publish/subscribe mechanism to transmit data to other replicas

2. The log is used as a consistency mechanism to order the updates that are applied to

multiple replicas

Somehow, perhaps because of this origin in database internals, the concept of a machine

readable log is not widely known, although, as we will see, this abstraction is ideal for

supporting all kinds of messaging, data flow, and real-time data processing.

Logs in Distributed Systems

The same problems that databases solve with logs (like distributing data to replicas and

agreeing on update order) are among the most fundamental problems for all distributed

systems.

The log-centric approach to distributed systems arises from a simple observation that I

will call the state machine replication principle:

If two identical, deterministic processes begin in the same state and get the same inputs

in the same order, they will produce the same output and end in the same state.

This may seem a bit obtuse, so let’s dive in and understand what it means.

Deterministic means that the processing isn’t timing dependent and doesn’t let any other

out-of-band input influence its results. For example, the following can be modeled as

nondeterministic: a multithreaded program whose output depends on the order of

execution of threads, or a program that makes decisions based on the results of a call to

gettimeofday(), or some other non-repeatable source of input. Of course, whether

these things are in fact truly deterministic is more a question about the foundations of

physics. However, for our purposes it is fine that we don’t know enough about their state

and inputs to model their output as a proper mathematical function.

The state of the process is whatever data remains on the machine, either in memory or on

disk, after our processing.

The part about getting the same input in the same order should ring a bell — that is where

the log comes in.

So this is actually a very intuitive notion: if you feed two deterministic pieces of code the

same input log, they will produce the same output in the same order.

The application to distributed computing is pretty obvious. You can reduce the problem of

making multiple machines all do the same thing to the problem of implementing a

consistent log to feed input to these processes. The purpose of the log here is to squeeze

all the nondeterminism out of the input stream to ensure that each replica that is

processing this input stays in sync.

Once you understand it, there is nothing complicated or deep about this principle: it

simply amounts to saying “deterministic processing is deterministic.” Nevertheless, I think

it is one of the more general tools for distributed systems design.

Nor is there anything new about this. If distributed computing is old enough to have a

classical approach, this would be it.

However, the implications of this basic design pattern are not that widely appreciated, and

the applications to enterprise architecture are appreciated even less.

One of the beautiful things about this is that the discrete log entry numbers now act as a

clock for the state of the replicas — you can describe the state of each replica by a single

number: the timestamp for the maximum log entry that it has processed. Two replicas at

the same time will be in the same state. Thus, this timestamp combined with the log

uniquely capture the entire state of the replica. This gives a discrete, event-driven notion

剩余83页未读，继续阅读

天降扫把星

粉丝: 4
资源: 11

Jay Kreps著《I Heart Logs》：事件数据与流处理指南

iHeart心电

I Heart Logs Event Data, Stream Processing, and Data Integration

Designing.Data-Intensive.Applications-2017

ApacheKafka入门介绍.pdf

团购风险博弈分析.pdf

Allen-Gale模型.pdf

KMRW声誉模型[定义].pdf

amccool.github.io：IO页面

jordenkreps.com:一个关于我和其他东西的网站..

Kafka 的简要介绍.docx

最新资源