设计大数据应用：可靠、可扩展与可维护系统的关键

需积分: 19 57 浏览量更新于2024-07-18 收藏 7.63MB PDF 举报

"《设计数据密集应用》是马丁·科莱博曼(Martin Kleppmann)撰写的一本关于构建可靠、可扩展和可维护的数据密集型应用的权威著作。这本书在2017年首次出版，可在GitHub上找到中文翻译版本。书中探讨了数据系统的基础、可靠性、可扩展性和可维护性等核心概念，并深入讨论了数据模型和查询语言的选择。" 在《设计数据密集应用》中，作者马丁·科莱博曼首先提出了构建高效数据系统的三大支柱：可靠性、可扩展性和可维护性。他强调，对于任何数据密集型应用来说，这些特性至关重要。 1. 可靠性：书中提到，可靠性不仅涉及硬件故障，还包括软件错误和人为错误。作者分析了这些因素对系统稳定性的影响，并探讨了为什么可靠性对于许多应用程序是至关重要的。他讨论了如何通过冗余、错误检测和恢复策略来提高系统的健壮性。 2. 可扩展性：科莱博曼解释了如何描述负载和性能，以及处理高负载的不同策略，如水平扩展和垂直扩展。他探讨了如何设计能够随着需求增长而扩展的系统，以确保持续的高性能。 3. 可维护性：这部分内容涵盖了操作简便（Operability）、简化复杂性（Simplicity）和适应变化的能力（Evolvability）。作者强调了设计易于操作和管理的系统的重要性，以及如何通过模块化和灵活的设计来降低复杂性，并允许系统随着业务需求的变化而演进。此外，书中还深入探讨了数据模型和查询语言的选择，对比了关系模型与文档模型，讲述了它们各自的优缺点。作者还可能讨论了SQL与NoSQL数据库，以及在不同场景下如何选择合适的模型和语言，以满足特定的应用需求。《设计数据密集应用》作为一本分布式系统必读书目，适合IT专业人士，尤其是对数据存储、大数据处理和系统架构感兴趣的开发者、工程师和架构师阅读。它提供了丰富的案例分析和实用建议，帮助读者理解并掌握构建大规模数据系统的关键原则和技术。

But reality is not that simple. There are many database systems with different charac‐

teristics, because different applications have different requirements. There are vari‐

ous approaches to caching, several ways of building search indexes, and so on. When

building an application, we still need to figure out which tools and which approaches

are the most appropriate for the task at hand. And it can be hard to combine tools

when you need to do something that a single tool cannot do alone.

This book is a journey through both the principles and the practicalities of data sys‐

tems, and how you can use them to build data-intensive applications. We will explore

what different tools have in common, what distinguishes them, and how they achieve

their characteristics.

In this chapter, we will start by exploring the fundamentals of what we are trying to

achieve: reliable, scalable, and maintainable data systems. We’ll clarify what those

things mean, outline some ways of thinking about them, and go over the basics that

we will need for later chapters. In the following chapters we will continue layer by

layer, looking at different design decisions that need to be considered when working

on a data-intensive application.

Thinking About Data Systems

We typically think of databases, queues, caches, etc. as being very different categories

of tools. Although a database and a message queue have some superficial similarity—

both store data for some time—they have very different access patterns, which means

different performance characteristics, and thus very different implementations.

So why should we lump them all together under an umbrella term like data systems?

Many new tools for data storage and processing have emerged in recent years. They

are optimized for a variety of different use cases, and they no longer neatly fit into

traditional categories [1]. For example, there are datastores that are also used as mes‐

sage queues (Redis), and there are message queues with database-like durability guar‐

antees (Apache Kafka). The boundaries between the categories are becoming blurred.

Secondly, increasingly many applications now have such demanding or wide-ranging

requirements that a single tool can no longer meet all of its data processing and stor‐

age needs. Instead, the work is broken down into tasks that can be performed effi‐

ciently on a single tool, and those different tools are stitched together using

application code.

For example, if you have an application-managed caching layer (using Memcached

or similar), or a full-text search server (such as Elasticsearch or Solr) separate from

your main database, it is normally the application code’s responsibility to keep those

caches and indexes in sync with the main database. Figure 1-1 gives a glimpse of what

this may look like (we will go into detail in later chapters).

4 | Chapter 1: Reliable, Scalable, and Maintainable Applications

In this book, we focus on three concerns that are important in most software systems:

Reliability

The system should continue to work correctly (performing the correct function at

the desired level of performance) even in the face of adversity (hardware or soft‐

ware faults, and even human error). See “Reliability” on page 6.

Scalability

As the system grows (in data volume, traffic volume, or complexity), there should

be reasonable ways of dealing with that growth. See “Scalability” on page 10.

Maintainability

Over time, many different people will work on the system (engineering and oper‐

ations, both maintaining current behavior and adapting the system to new use

cases), and they should all be able to work on it productively. See “Maintainabil‐

ity” on page 18.

These words are often cast around without a clear understanding of what they mean.

In the interest of thoughtful engineering, we will spend the rest of this chapter

exploring ways of thinking about reliability, scalability, and maintainability. Then, in

the following chapters, we will look at various techniques, architectures, and algo‐

rithms that are used in order to achieve those goals.

Reliability

Everybody has an intuitive idea of what it means for something to be reliable or unre‐

liable. For software, typical expectations include:

• The application performs the function that the user expected.

• It can tolerate the user making mistakes or using the software in unexpected

ways.

• Its performance is good enough for the required use case, under the expected

load and data volume.

• The system prevents any unauthorized access and abuse.

If all those things together mean “working correctly,” then we can understand relia‐

bility as meaning, roughly, “continuing to work correctly, even when things go

wrong.”

The things that can go wrong are called faults, and systems that anticipate faults and

can cope with them are called fault-tolerant or resilient. The former term is slightly

misleading: it suggests that we could make a system tolerant of every possible kind of

fault, which in reality is not feasible. If the entire planet Earth (and all servers on it)

were swallowed by a black hole, tolerance of that fault would require web hosting in

6 | Chapter 1: Reliable, Scalable, and Maintainable Applications

space—good luck getting that budget item approved. So it only makes sense to talk

about tolerating certain types of faults.

Note that a fault is not the same as a failure [2]. A fault is usually defined as one com‐

ponent of the system deviating from its spec, whereas a failure is when the system as a

whole stops providing the required service to the user. It is impossible to reduce the

probability of a fault to zero; therefore it is usually best to design fault-tolerance

mechanisms that prevent faults from causing failures. In this book we cover several

techniques for building reliable systems from unreliable parts.

Counterintuitively, in such fault-tolerant systems, it can make sense to increase the

rate of faults by triggering them deliberately—for example, by randomly killing indi‐

vidual processes without warning. Many critical bugs are actually due to poor error

handling [3]; by deliberately inducing faults, you ensure that the fault-tolerance

machinery is continually exercised and tested, which can increase your confidence

that faults will be handled correctly when they occur naturally. The Netflix Chaos

Monkey [4] is an example of this approach.

Although we generally prefer tolerating faults over preventing faults, there are cases

where prevention is better than cure (e.g., because no cure exists). This is the case

with security matters, for example: if an attacker has compromised a system and

gained access to sensitive data, that event cannot be undone. However, this book

mostly deals with the kinds of faults that can be cured, as described in the following

sections.

Hardware Faults

When we think of causes of system failure, hardware faults quickly come to mind.

Hard disks crash, RAM becomes faulty, the power grid has a blackout, someone

unplugs the wrong network cable. Anyone who has worked with large datacenters

can tell you that these things happen all the time when you have a lot of machines.

Hard disks are reported as having a mean time to failure (MTTF) of about 10 to 50

years [5, 6]. Thus, on a storage cluster with 10,000 disks, we should expect on average

one disk to die per day.

Our first response is usually to add redundancy to the individual hardware compo‐

nents in order to reduce the failure rate of the system. Disks may be set up in a RAID

configuration, servers may have dual power supplies and hot-swappable CPUs, and

datacenters may have batteries and diesel generators for backup power. When one

component dies, the redundant component can take its place while the broken com‐

ponent is replaced. This approach cannot completely prevent hardware problems

from causing failures, but it is well understood and can often keep a machine running

uninterrupted for years.

Reliability | 7

i. Defined in “Approaches for Coping with Load” on page 17.

Until recently, redundancy of hardware components was sufficient for most applica‐

tions, since it makes total failure of a single machine fairly rare. As long as you can

restore a backup onto a new machine fairly quickly, the downtime in case of failure is

not catastrophic in most applications. Thus, multi-machine redundancy was only

required by a small number of applications for which high availability was absolutely

essential.

However, as data volumes and applications’ computing demands have increased,

more applications have begun using larger numbers of machines, which proportion‐

ally increases the rate of hardware faults. Moreover, in some cloud platforms such as

Amazon Web Services (AWS) it is fairly common for virtual machine instances to

become unavailable without warning [7], as the platforms are designed to prioritize

flexibility and elasticity

over single-machine reliability.

Hence there is a move toward systems that can tolerate the loss of entire machines, by

using software fault-tolerance techniques in preference or in addition to hardware

redundancy. Such systems also have operational advantages: a single-server system

requires planned downtime if you need to reboot the machine (to apply operating

system security patches, for example), whereas a system that can tolerate machine

failure can be patched one node at a time, without downtime of the entire system (a

rolling upgrade; see Chapter 4).

Software Errors

We usually think of hardware faults as being random and independent from each

other: one machine’s disk failing does not imply that another machine’s disk is going

to fail. There may be weak correlations (for example due to a common cause, such as

the temperature in the server rack), but otherwise it is unlikely that a large number of

hardware components will fail at the same time.

Another class of fault is a systematic error within the system [8]. Such faults are

harder to anticipate, and because they are correlated across nodes, they tend to cause

many more system failures than uncorrelated hardware faults [5]. Examples include:

• A software bug that causes every instance of an application server to crash when

given a particular bad input. For example, consider the leap second on June 30,

2012, that caused many applications to hang simultaneously due to a bug in the

Linux kernel [9].

• A runaway process that uses up some shared resource—CPU time, memory, disk

space, or network bandwidth.

8 | Chapter 1: Reliable, Scalable, and Maintainable Applications

剩余584页未读，继续阅读

rocketluoqq

粉丝: 2
资源: 4

设计大数据应用：可靠、可扩展与可维护系统的关键

设计数据密集型应用

设计数据密集应用：可信赖、可扩展与维护的基石

设计数据密集型应用.zip

设计数据密集型应用.pdf

<设计数据密集型应用>

设计数据密集型应用(2017)

设计数据密集型应用（初翻）

设计数据密集型应用(影印英文原版)

设计数据密集型应用的核心理念

设计数据密集型应用：中文版解读

最新资源