构建可靠、可扩展的数据密集型应用：马丁·克莱普曼的深度解析

需积分: 13 90 浏览量更新于2024-07-18 收藏 5.15MB PDF 举报

《设计数据密集型应用》(Designing Data-Intensive Applications)是由马丁·克莱普曼(Martin Kleppmann)编著的一本经典分布式学习书籍，着重探讨了构建可靠、可扩展和易于维护的系统背后的大理念。该书于2017年3月首次出版，适用于那些在处理海量数据和高并发场景下的开发者和架构师。在第一部分"数据系统的基础"中，作者强调了系统的可靠性（包括硬件故障、软件错误和人为失误）对整体性能的影响。他深入分析了可靠性的重要性，并讨论了如何通过冗余备份、错误处理和恢复机制来提高系统的稳定性。章节中还提到了如何衡量负载（如吞吐量和响应时间），以及应对负载增长的各种策略，如水平扩展和垂直扩展。接下来，作者将焦点转向数据模型和查询语言。他比较了关系模型与文档模型的优缺点，指出选择哪种模型取决于具体的应用场景和需求。这包括了对SQL和NoSQL等不同查询语言特性的探讨，以及它们在处理不同类型数据集时的适用性。在“数据模型与查询语言”这一章中，克莱普曼教授读者如何选择合适的数据结构来支持高效查询，同时保持数据一致性。他还可能介绍了如何设计查询优化算法，以确保在大规模数据集上执行查询时的性能。除此之外，书中还涵盖了维护性和操作性这两个关键主题。通过简洁的设计和良好的模块化原则，作者提倡使系统的维护变得更加容易。同时，为了便于运维人员的工作，书中可能包含了一些自动化运维工具和监控系统的介绍。此外，书中还会讨论如何使系统具备进化能力，以便在需求变化时能够轻松地进行调整和扩展。这涉及到了版本控制、持续集成/持续部署(CI/CD)流程以及架构设计中的灵活性。总结来说，《设计数据密集型应用》不仅提供了理论知识，还包含了实用的指导，帮助读者理解如何在现代IT环境中设计出高可用、高性能且易于管理的数据系统。无论是对于初入行业的开发者，还是经验丰富的架构师，这本书都是一个宝贵的学习资源，帮助他们在处理大数据挑战时做出明智的决策。

But reality is not that simple. There are many database systems with different charac‐

teristics, because different applications have different requirements. There are vari‐

ous approaches to caching, several ways of building search indexes, and so on. When

building an application, we still need to figure out which tools and which approaches

are the most appropriate for the task at hand. And it can be hard to combine tools

when you need to do something that a single tool cannot do alone.

This book is a journey through both the principles and the practicalities of data sys‐

tems, and how you can use them to build data-intensive applications. We will explore

what different tools have in common, what distinguishes them, and how they achieve

their characteristics.

In this chapter, we will start by exploring the fundamentals of what we are trying to

achieve: reliable, scalable, and maintainable data systems. We’ll clarify what those

things mean, outline some ways of thinking about them, and go over the basics that

we will need for later chapters. In the following chapters we will continue layer by

layer, looking at different design decisions that need to be considered when working

on a data-intensive application.

Thinking About Data Systems

We typically think of databases, queues, caches, etc. as being very different categories

of tools. Although a database and a message queue have some superficial similarity—

both store data for some time—they have very different access patterns, which means

different performance characteristics, and thus very different implementations.

So why should we lump them all together under an umbrella term like data systems?

Many new tools for data storage and processing have emerged in recent years. They

are optimized for a variety of different use cases, and they no longer neatly fit into

traditional categories [1]. For example, there are datastores that are also used as mes‐

sage queues (Redis), and there are message queues with database-like durability guar‐

antees (Apache Kafka). The boundaries between the categories are becoming blurred.

Secondly, increasingly many applications now have such demanding or wide-ranging

requirements that a single tool can no longer meet all of its data processing and stor‐

age needs. Instead, the work is broken down into tasks that can be performed effi‐

ciently on a single tool, and those different tools are stitched together using

application code.

For example, if you have an application-managed caching layer (using Memcached

or similar), or a full-text search server (such as Elasticsearch or Solr) separate from

your main database, it is normally the application code’s responsibility to keep those

caches and indexes in sync with the main database. Figure 1-1 gives a glimpse of what

this may look like (we will go into detail in later chapters).

4 | Chapter 1: Reliable, Scalable, and Maintainable Applications

In this book, we focus on three concerns that are important in most software systems:

Reliability

The system should continue to work correctly (performing the correct function at

the desired level of performance) even in the face of adversity (hardware or soft‐

ware faults, and even human error). See “Reliability” on page 6.

Scalability

As the system grows (in data volume, traffic volume, or complexity), there should

be reasonable ways of dealing with that growth. See “Scalability” on page 10.

Maintainability

Over time, many different people will work on the system (engineering and oper‐

ations, both maintaining current behavior and adapting the system to new use

cases), and they should all be able to work on it productively. See “Maintainabil‐

ity” on page 18.

These words are often cast around without a clear understanding of what they mean.

In the interest of thoughtful engineering, we will spend the rest of this chapter

exploring ways of thinking about reliability, scalability, and maintainability. Then, in

the following chapters, we will look at various techniques, architectures, and algo‐

rithms that are used in order to achieve those goals.

Reliability

Everybody has an intuitive idea of what it means for something to be reliable or unre‐

liable. For software, typical expectations include:

• The application performs the function that the user expected.

• It can tolerate the user making mistakes or using the software in unexpected

ways.

• Its performance is good enough for the required use case, under the expected

load and data volume.

• The system prevents any unauthorized access and abuse.

If all those things together mean “working correctly,” then we can understand relia‐

bility as meaning, roughly, “continuing to work correctly, even when things go

wrong.”

The things that can go wrong are called faults, and systems that anticipate faults and

can cope with them are called fault-tolerant or resilient. The former term is slightly

misleading: it suggests that we could make a system tolerant of every possible kind of

fault, which in reality is not feasible. If the entire planet Earth (and all servers on it)

were swallowed by a black hole, tolerance of that fault would require web hosting in

6 | Chapter 1: Reliable, Scalable, and Maintainable Applications

space—good luck getting that budget item approved. So it only makes sense to talk

about tolerating certain types of faults.

Note that a fault is not the same as a failure [2]. A fault is usually defined as one com‐

ponent of the system deviating from its spec, whereas a failure is when the system as a

whole stops providing the required service to the user. It is impossible to reduce the

probability of a fault to zero; therefore it is usually best to design fault-tolerance

mechanisms that prevent faults from causing failures. In this book we cover several

techniques for building reliable systems from unreliable parts.

Counterintuitively, in such fault-tolerant systems, it can make sense to increase the

rate of faults by triggering them deliberately—for example, by randomly killing indi‐

vidual processes without warning. Many critical bugs are actually due to poor error

handling [3]; by deliberately inducing faults, you ensure that the fault-tolerance

machinery is continually exercised and tested, which can increase your confidence

that faults will be handled correctly when they occur naturally. The Netflix Chaos

Monkey [4] is an example of this approach.

Although we generally prefer tolerating faults over preventing faults, there are cases

where prevention is better than cure (e.g., because no cure exists). This is the case

with security matters, for example: if an attacker has compromised a system and

gained access to sensitive data, that event cannot be undone. However, this book

mostly deals with the kinds of faults that can be cured, as described in the following

sections.

Hardware Faults

When we think of causes of system failure, hardware faults quickly come to mind.

Hard disks crash, RAM becomes faulty, the power grid has a blackout, someone

unplugs the wrong network cable. Anyone who has worked with large datacenters

can tell you that these things happen all the time when you have a lot of machines.

Hard disks are reported as having a mean time to failure (MTTF) of about 10 to 50

years [5, 6]. Thus, on a storage cluster with 10,000 disks, we should expect on average

one disk to die per day.

Our first response is usually to add redundancy to the individual hardware compo‐

nents in order to reduce the failure rate of the system. Disks may be set up in a RAID

configuration, servers may have dual power supplies and hot-swappable CPUs, and

datacenters may have batteries and diesel generators for backup power. When one

component dies, the redundant component can take its place while the broken com‐

ponent is replaced. This approach cannot completely prevent hardware problems

from causing failures, but it is well understood and can often keep a machine running

uninterrupted for years.

Reliability | 7

i. Defined in “Approaches for Coping with Load” on page 17.

Until recently, redundancy of hardware components was sufficient for most applica‐

tions, since it makes total failure of a single machine fairly rare. As long as you can

restore a backup onto a new machine fairly quickly, the downtime in case of failure is

not catastrophic in most applications. Thus, multi-machine redundancy was only

required by a small number of applications for which high availability was absolutely

essential.

However, as data volumes and applications’ computing demands have increased,

more applications have begun using larger numbers of machines, which proportion‐

ally increases the rate of hardware faults. Moreover, in some cloud platforms such as

Amazon Web Services (AWS) it is fairly common for virtual machine instances to

become unavailable without warning [7], as the platforms are designed to prioritize

flexibility and elasticity

over single-machine reliability.

Hence there is a move toward systems that can tolerate the loss of entire machines, by

using software fault-tolerance techniques in preference or in addition to hardware

redundancy. Such systems also have operational advantages: a single-server system

requires planned downtime if you need to reboot the machine (to apply operating

system security patches, for example), whereas a system that can tolerate machine

failure can be patched one node at a time, without downtime of the entire system (a

rolling upgrade; see Chapter 4).

Software Errors

We usually think of hardware faults as being random and independent from each

other: one machine’s disk failing does not imply that another machine’s disk is going

to fail. There may be weak correlations (for example due to a common cause, such as

the temperature in the server rack), but otherwise it is unlikely that a large number of

hardware components will fail at the same time.

Another class of fault is a systematic error within the system [8]. Such faults are

harder to anticipate, and because they are correlated across nodes, they tend to cause

many more system failures than uncorrelated hardware faults [5]. Examples include:

•

A software bug that causes every instance of an application server to crash when

given a particular bad input. For example, consider the leap second on June 30,

2012, that caused many applications to hang simultaneously due to a bug in the

Linux kernel [9].

• A runaway process that uses up some shared resource—CPU time, memory, disk

space, or network bandwidth.

8 | Chapter 1: Reliable, Scalable, and Maintainable Applications

剩余584页未读，继续阅读

WalkingInTheWind

粉丝: 1110
资源: 8

构建可靠、可扩展的数据密集型应用：马丁·克莱普曼的深度解析

Designing Data-Intensive Application-cn.pdf

Designing Data-Intensive Applications pdf

Designing Data-Intensive Applications 影印版

Designing Data-Intensive Applications 中文版

Designing Data-Intensive Applications 英文 pdf

Designing Data-Intensive Applications_软件架构_

designing data-intensive applications pdf

designing data-intensive applications awz3 mobi

基于freeRTOS和STM32F103x的手机远程控制浴室温度系统设计源码

LABVIEW程序实例-web写数据.zip

最新资源