《I Heart Logs》：数据流基石与大规模系统治理指南

需积分: 33 5 浏览量更新于2024-07-19 收藏 4.72MB PDF 举报

"I Heart Logs.pdf" 是一本由 Jay Kreps 编写的关于日志管理的深入指南，该书在 IT 行业中备受瞩目。它强调了日志在现代数据存储和处理系统中的核心地位，即使在NoSQL数据库和加密货币等技术日益复杂的背景下，日志仍然发挥着至关重要的作用。作为一本实用的资源，它旨在帮助读者理解如何将日志视为基础设施的关键组成部分，从而实现数据的可靠流动和大规模数据处理的效率。本书涵盖了以下几个关键主题： 1. 日志的重要性：日志并非仅仅是一种记录事件的简单工具，而是许多复杂系统背后隐藏的抽象层。它们是追踪系统行为、调试问题、监控性能以及实现数据集成的基础。 2. 事件数据和流处理：书中深入探讨了事件数据的收集和处理，特别是在实时数据处理场景中，如Apache Kafka 或其他流处理框架的应用。通过有效管理事件数据，可以优化系统的响应速度和可扩展性。 3. 数据集成与管理：在大数据时代，数据集成是不可或缺的一环。作者分享了如何利用日志来整合来自不同源头的数据，确保数据一致性，这对于构建数据驱动的产品和服务至关重要。 4. 设计原则与实践：书中提供了实用的设计原则和案例分析，帮助读者学习如何构建健壮的、可维护的数据流架构，使之适应未来数据密集型应用的需求。 5. 行业认可：这本书得到了业内专家的积极评价，如 Martin Kleppman（《Designing Data-Intensive Applications》作者）认为这是一本转折点之作，标志着数据系统设计的一个进步；而 Monica Rogati（Jawbone 数据VP）则强调了可靠的data flow对于数据产品开发和故事讲述的重要性。通过阅读"I Heart Logs.pdf"，读者不仅能掌握日志管理的最新趋势和技术，还能提升对现代IT基础设施的理解，从而在实际工作中更好地应对挑战，提升数据处理能力。无论你是服务器端开发者、数据工程师，还是寻求在大数据领域进一步发展的专业人士，这本书都将提供宝贵的知识和洞见。

Figure 1-1. An excerpt from an Apache log

The log I’ll be discussing is a little more general and closer to what in the database or

systems world might be called a commit log or journal. It is an append-only sequence

of records ordered by time, as in Figure 1-2.

Figure 1-2. A structured log (records are numbered beginning with 0 based on the order

in which they are written)

Each rectangle represents a record that was appended to the log. Records are stored

in the order they were appended. Reads proceed from left to right. Each entry

appended to the log is assigned a unique, sequential log entry number that acts as its

unique key. The contents and format of the records aren’t important for the purposes

of this discussion. To be concrete, we can just imagine each record to be a JSON blob,

but of course any data format will do.

The ordering of records defines a notion of “time” since entries to the left are defined

to be older then entries to the right. The log entry number can be thought of as the

“timestamp” of the entry. Describing this ordering as a notion of time seems a bit odd

at first, but it has the convenient property of being decoupled from any particular

physical clock. This property will turn out to be essential as we get to distributed sys‐

tems.

2 | Chapter 1: Introduction

www.allitebooks.com

A log is not all that different from a file or a table. A file is an array of bytes, a table is

an array of records, and a log is really just a kind of table or file where the records are

sorted by time.

You can see the analogy to the Apache log I showed earlier: both are append-only

sequences of records. However, it is important that we think about the log as an

abstract data structure, not a text file.

At this point you might be wondering, “Why is it worth talking about something so

simple?” How is an append-only sequence of records in any way related to data sys‐

tems? The answer is that logs have a specific purpose: they record what happened and

when. For distributed data systems this is, in many ways, the very heart of the

problem.

Logs in Databases

I don’t know where the log concept originated—it is probably one of those things like

binary search that is too simple for the inventor to realize it was an invention. It is

present as early as IBM’s System R. The usage in databases has to do with keeping in

sync a variety of data structures and indexes in the presence of crashes. To make this

atomic and durable, a database uses a log to write out information about the records

it will be modifying before applying the changes to all the various data structures that

it maintains. The log is the record of what happened, and each table or index is a pro‐

jection of this history into some useful data structure or index. Since the log is imme‐

diately persisted, it is used as the authoritative source in restoring all other persistent

structures in the event of a crash.

Over time, the usage of the log grew from an implementation detail of the ACID

database properties (atomicity, consistency, isolation, and durability) to a method for

replicating data between databases. It turns out that the sequence of changes that hap‐

pened on the database is exactly what is needed to keep a remote replica database in

sync. Oracle, MySQL, PostgreSQL, and MongoDB include log shipping protocols to

transmit portions of a log to replica databases that act as slaves. The slaves can then

apply the changes recorded in the log to their own local data structures to stay in sync

with the master. Oracle has productized the log as a general data subscription mecha‐

nism for non-Oracle data subscribers with their XStreams and GoldenGate products,

and similar facilities exist in MySQL and PostgreSQL.

In fact, the use of logs in much of the rest of this book will be variations on the two

uses in database internals:

1. The log is used as a publish/subscribe mechanism to transmit data to other

replicas

Logs in Databases | 3

2. The log is used as a consistency mechanism to order the updates that are applied

to multiple replicas

Somehow, perhaps because of this origin in database internals, the concept of a

machine readable log is not widely known, although, as we will see, this abstraction is

ideal for supporting all kinds of messaging, data flow, and real-time data processing.

Logs in Distributed Systems

The same problems that databases solve with logs (like distributing data to replicas

and agreeing on update order) are among the most fundamental problems for all dis‐

tributed systems.

The log-centric approach to distributed systems arises from a simple observation that

I will call the state machine replication principle:

If two identical, deterministic processes begin in the same state and get the same inputs

in the same order, they will produce the same output and end in the same state.

This may seem a bit obtuse, so let’s dive in and understand what it means.

Deterministic means that the processing isn’t timing dependent and doesn’t let any

other out-of-band input influence its results. For example, the following can be mod‐

eled as nondeterministic: a multithreaded program whose output depends on the

order of execution of threads, or a program that makes decisions based on the results

of a call to gettimeofday(), or some other non-repeatable source of input. Of course,

whether these things are in fact truly deterministic is more a question about the foun‐

dations of physics. However, for our purposes it is fine that we don’t know enough

about their state and inputs to model their output as a proper mathematical function.

The state of the process is whatever data remains on the machine, either in memory

or on disk, after our processing.

The part about getting the same input in the same order should ring a bell—that is

where the log comes in.

So this is actually a very intuitive notion: if you feed two deterministic pieces of code

the same input log, they will produce the same output in the same order.

The application to distributed computing is pretty obvious. You can reduce the prob‐

lem of making multiple machines all do the same thing to the problem of implement‐

ing a consistent log to feed input to these processes. The purpose of the log here is to

squeeze all the nondeterminism out of the input stream to ensure that each replica

that is processing this input stays in sync.

Once you understand it, there is nothing complicated or deep about this principle: it

simply amounts to saying “deterministic processing is deterministic.” Nevertheless, I

think it is one of the more general tools for distributed systems design.

4 | Chapter 1: Introduction

剩余58页未读，继续阅读

nikitar

粉丝: 6
资源: 44

《I Heart Logs》：数据流基石与大规模系统治理指南

iHeart心电

I Heart Logs Event Data, Stream Processing, and Data Integration

logs.wxml

logs.js

logs.zip

logs.pcapng

stream.logs.zip

Apache.Logs.Viewer.v.5.39.Patch-URET.zip

logs.pcapng解析.docx

WorkLogs.docx

最新资源