构建数据密集型应用：可靠、可扩展和可维护的系统设计

5星 · 超过95%的资源需积分: 16 178 浏览量更新于2024-07-22 7 收藏 5.62MB PDF 举报

"Designing Data-Intensive Applications" 是一本由 Martin Kleppmann 撰写的书籍，专注于探讨如何构建可靠、可扩展和易于维护的数据密集型应用。本书旨在帮助软件工程师和架构师理解如何在长期内设计出高效的数据系统，并通过各种流行软件包和框架的内部工作原理作为实例来阐述关键原则、算法和权衡。本书分为三个部分： Part I. The Big Picture 1. Reliable, Scalable and Maintainable Applications：这部分讨论了在构建大型数据系统时应考虑的关键因素，包括可靠性、可扩展性和可维护性。作者会介绍如何通过设计来确保系统的高可用性和容错性，以及如何在性能和复杂性之间找到平衡。 2. The Battle of the Data Models：这一章深入探讨不同的数据模型，如关系型数据库、NoSQL 数据库、图数据库等。它分析了每种模型的优缺点，以及在不同场景下如何选择合适的数据模型。 3. Storage and Retrieval：讨论了数据存储和检索的策略，包括索引、查询优化和数据分布，以及如何有效地处理大量数据的读写操作。 Part II. Systems of Record 4. Replication：这部分详细介绍了数据复制技术，这是实现高可用性和容错性的关键。讨论了主从复制、多活复制等不同复制模式，以及它们在解决一致性问题上的角色。这本书不仅关注理论，还强调实践，提供了很多实际案例，帮助读者理解和应用这些概念。通过对当前工具和技术的分析，读者将学会如何根据需求选择合适的工具，并理解如何组合不同的组件来构建一个强大的应用架构。此外，书中还包括如何建立对系统的洞察力，以便更有效地诊断和解决问题。通过阅读《Designing Data-Intensive Applications》，读者可以提升自己在大数据处理、分布式系统和可靠性设计方面的专业能力，为构建下一个级别的应用程序打下坚实基础。这本书是任何希望在数据密集型应用领域深化理解的开发者或架构师的重要参考资源。

reboot the machine (to apply security patches, for example), whereas a system that can tolerate

machine failure can be patched one node at a time, without downtime of the entire system.

Software errors

We usually think of hardware faults as being random and independent from each other: one

machine’s disk failing does not imply that another machine’s disk is going to fail. There may be

weak correlations (for example due to a common cause, such as the temperature in the server

rack), but otherwise it is unlikely that a large number of hardware components will fail at the

same time.

Another class of fault is a systematic error within the system. Such faults are harder to anticipate,

and because they are correlated across nodes, they tend to cause many more system failures

than uncorrelated hardware faults [

[4]

]. Examples include:



A software bug that causes every instance of an application server to crash when given a particular bad

input. For example, consider the leap second on June 30, 2012 that caused many applications to hang

simultaneously, due to a bug in the Linux kernel. [

[6]

]



A runaway process uses up some shared resource—CPU time, memory, disk space or network

bandwidth.

 A service that the system depends on slows down, becomes unresponsive or starts returning corrupted

responses.

 Cascading failures, where a small fault in one component triggers a fault in another component, which

in turn triggers further faults (see [

[7]

] for example).

The bugs that cause these kinds of software fault often lie dormant for a long time until they are

triggered by an unusual set of circumstances. In those circumstances, it is revealed that the

software is making some kind of assumption about its environment—and whilst that assumption

is usually true, it eventually stops being true for some reason.

There is no quick solution to the problem of systematic faults in software. Lots of small things

can help: carefully thinking about assumptions and interactions in the system, thorough testing,

measuring, monitoring and analyzing system behavior in production. If a system is expected to

provide some guarantee (for example, in a message queue, that the number of incoming

messages equals the number of outgoing messages), it can constantly check itself while it is

running, and raise an alert if a discrepancy is found [

[8]

Human errors

Humans design and build software systems, and the operators who keep the system running are

also human. Even when they have the best intentions, humans are known to be unreliable. How

do we make our system reliable, in spite of unreliable humans?

The best systems combine several approaches:

 Design systems in a way that minimizes opportunities for error. For example, well-designed

abstractions, APIs and admin interfaces make it easy to do “the right thing”, and discourage “the

wrong thing”. However, if the interfaces are too restrictive, people will work around them, negating

their benefit, so this is a tricky balance to get right.

 Decouple the places where people make the most mistakes from the places where they can cause

failures. In particular, provide fully-featured non-production sandbox environments where people can

explore and experiment safely, using real data, without affecting real users.

 Test thoroughly at all levels, from unit tests to whole-system integration tests and manual tests.

Automated testing is widely used, well understood, and especially valuable for covering corner cases

that rarely arise in normal operation.

 Allow quick and easy recovery from human errors, to minimize the impact in the case of a failure. For

example, make it fast to roll back configuration changes, roll out new code gradually (so that any

unexpected bugs affect only a small subset of users), and provide tools to recompute data (in case it

turns out that the old computation was incorrect).



Set up detailed and clear monitoring and telemetry, such as performance metrics and error rates. (The

term telemetry is not often applied to software, but it is very apt. We can learn about good telemetry

from other engineering disciplines, such as aerospace [

[9]

].) Monitoring can show us early warning

signals, and allow us to check whether any assumptions or constraints are being violated. When a

problem occurs, metrics can be invaluable in diagnosing the issue.

 Good management practices, training etc—a complex and important aspect, and beyond the scope of

this book.

How important is reliability?

Reliability is not just for nuclear power stations and air traffic control software—more mundane

applications are also expected to work reliably. Bugs in business applications cause lost

productivity (and legal risks if figures are reported incorrectly), and outages of e-commerce sites

can have huge costs in terms of lost revenue and reputation.

Even in “non-critical” applications we have a responsibility to our users. Consider a parent who

stores all pictures and videos of their children in your photo application [

[10]

]. How would they

feel if that database was suddenly corrupted? Would they know how to restore it from a backup?

There are situations in which we may choose to sacrifice reliability in order to reduce

development cost (e.g. when developing a prototype product for an unproven market) or

operational cost (e.g. for a service with a very narrow profit margin)—but we should be very

conscious of when we are cutting corners.

Scalability

Even if a system is working reliably today, that doesn’t mean it will necessarily work reliably in

future. One common reason for degradation is increased load: perhaps it has grown from

10,000 concurrent users to 100,000 concurrent users, or from 1 million to 10 million. Perhaps it

is processing much larger volumes of data than it did before.

Scalability is the term we use to describe a system’s ability to adapt to increased load. Note,

however, that it is not a one-dimensional label that we can attach to a system: it is meaningless

to say “X is scalable” or “Y doesn’t scale”. Rather, discussing scalability means to discuss the

question: if the system grows in a particular way, what are our options for coping with the

growth?

Describing load

First, we need to succinctly describe the current load on the system; only then can we discuss

growth questions (what happens if our load doubles?). Load can be described with a few

numbers which we call load parameters. The best choice of parameters depends on the the

architecture of your system: perhaps it’s requests per second, ratio of reads to writes, the

number of simultaneously active users, or something else. Perhaps the average case is what

matters for you, or perhaps your bottleneck is dominated by a small number of extreme cases.

To make this idea more concrete, let’s consider Twitter as an example, using data published in

November 2012 [

[11]

]. Two of Twitter’s main operations are:

Post tweet

A user can publish a new message to their followers (4.6 k requests/sec on average, over 12 k

requests/sec at peak).

Home timeline

A user can view tweets recently published by the people they follow (300 k requests/sec).

Simply handling 12,000 writes per second (the peak rate for posting tweets) would be fairly easy.

However, Twitter’s scaling challenge is not primarily due to tweet volume, but due to

fan-out

[12]

—each user follows many people, and each user is followed by many people. There are

broadly two approaches to implementing these two operations:

timeline reads, and so in this case it’s preferable to do more work at write time and less at read

time.

However, the downside of approach 2 is that posting a tweet now requires a lot of extra work.

On average, a tweet is delivered to about 75 followers, so 4.6 k tweets per second become 345 k

writes per second to the home timeline caches. But this average hides the fact that the number

of followers per user varies wildly, and some users have over 30 million followers. This means

that a single tweet may result in over 30 million writes to home timelines! Doing this in a timely

manner—Twitter tries to deliver tweets to followers within 5 seconds—is a significant challenge.

In the example of Twitter, the distribution of followers per user (maybe weighted by how often

those users tweet), is a key load parameter for discussing scalability, since it determines the

fan-out load. Your application may have very different characteristics, but you can apply similar

principles to reasoning about its load.

The final twist of the Twitter anecdote: now that approach 2 is robustly implemented, Twitter is

moving to a hybrid of both approaches. Most users’ tweets continue to be fanned out to home

timelines at the time when they are posted, but a small number of users with a very large

number of followers are excepted from this fan-out. Instead, when the home timeline is read,

the tweets from celebrities followed by the user are fetched separately and merged with the

home timeline when the timeline is read, like in approach 1. This hybrid approach is able to

deliver consistently good performance.

Describing performance

Once you have described the load on our system, you can investigate what happens when the

load increases. You can look at it in two ways:



When you increase a load parameter, and keep the system resources (CPU, memory, network

bandwidth, etc.) unchanged, how is performance of your system affected?

 When you increase a load parameter, how much do you need to increase the resources if you want to

keep performance unchanged?

Both questions require performance numbers, so let’s look briefly at describing the performance

of a system.

In a batch-processing system such as Hadoop, we usually care about throughput—the number of

records we can process per second, or the total time it takes to run a job on a dataset of a certain

size.

[13]

In online systems, latency is usually more important—the time it takes to serve a request,

also known as response time.

剩余143页未读，继续阅读

ramissue

粉丝: 354
资源: 1487

构建数据密集型应用：可靠、可扩展和可维护的系统设计

DesigningData-IntensiveApplications. ScalableandMaintainableSystem

design data intensive applications pdf

Designing Data-Intensive Applications 英文 pdf

Designing.Data-Intensive.Applications-2017

Designing.Data-Intensive.Applications 设计数据密集型应用（html中文版全网首发）

Designing-Data-Intensive-Applications

Designing-Data-Intensive-Applications 英文高清完整下载

designing-data-Intensive-applications:以数据为中心的应用设计研究

designing-data-intensive-applications-notes:有关设计数据密集型应用程序的说明（进行中）

java版飞机大战源码-designing-data-intensive-applications:设计数据密集型应用程序

最新资源