数据密集应用设计：构建可靠、可扩展与维护的系统全解析

需积分: 9 39 浏览量更新于2024-07-19 1 收藏 23.83MB PDF 举报

《设计数据密集型应用》(Designing Data-Intensive Applications) 是2017年由Martin Kleppmann撰写的一本备受推崇的IT技术书籍。本书旨在深入探讨在构建可靠、可扩展和易于维护的系统时所涉及的核心理念和技术。作者以其丰富的经验和深厚的专业知识，针对大数据、分布式计算、数据库和存储等领域，提供了实用的设计策略和最佳实践。书中涵盖的主题包括但不限于： 1. **设计理念**：Kleppmann阐述了设计数据密集型应用的基本原则，如容错性、一致性、可伸缩性和灵活性，强调了在处理海量数据时如何平衡这些关键要素。 2. **分布式系统**：书中详细讲解了分布式系统的设计模式，如复制、分片、分区等，以及如何通过分布式架构实现系统的高可用性和负载均衡。 3. **数据库设计**：读者可以了解到如何选择合适的数据库模型（关系型、NoSQL或图形数据库），以及优化查询性能、数据模型设计和数据分区等技术。 4. **存储管理**：涵盖了存储层的挑战，如数据压缩、缓存策略、数据持久化和备份恢复方法，以及如何应对不同存储技术（如Hadoop HDFS、Redis等）。 5. **故障处理与恢复**：书中讨论了如何设计能够容忍故障的系统，包括错误检测、隔离、恢复策略和监控工具的使用。 6. **性能优化**：涉及系统性能瓶颈分析、基准测试和调优技巧，帮助读者理解如何在不断变化的数据环境和用户需求下保持高效运行。 7. **实战案例分析**：书中穿插了真实世界的项目实例，展示了如何将理论知识应用于实际场景，帮助读者理解和掌握设计数据密集型应用的实战技巧。《设计数据密集型应用》不仅是IT专业人士的必备参考书，也是数据科学家、工程师和架构师提升技能，解决大规模数据处理问题的重要资源。无论是在构建云服务、大数据处理平台，还是在设计高性能互联网服务时，这本书都能提供深刻的见解和实用的指导。对于希望在这个领域有所建树的读者来说，这是一本不可多得的宝典。

which it is changing—as opposed to compute-intensive, where CPU cycles are the

bottleneck.

The tools and technologies that help data-intensive applications store and process

data have been rapidly adapting to these changes. New types of database systems

(“NoSQL”) have been getting lots of attention, but message queues, caches, search

indexes, frameworks for batch and stream processing, and related technologies are

very important too. Many applications use some combination of these.

The buzzwords that fill this space are a sign of enthusiasm for the new possibilities,

which is a great thing. However, as software engineers and architects, we also need to

have a technically accurate and precise understanding of the various technologies and

their trade-offs if we want to build good applications. For that understanding, we

have to dig deeper than buzzwords.

Fortunately, behind the rapid changes in technology, there are enduring principles

that remain true, no matter which version of a particular tool you are using. If you

understand those principles, you’re in a position to see where each tool fits in, how to

make good use of it, and how to avoid its pitfalls. That’s where this book comes in.

The goal of this book is to help you navigate the diverse and fast-changing landscape

of technologies for processing and storing data. This book is not a tutorial for one

particular tool, nor is it a textbook full of dry theory. Instead, we will look at examples

of successful data systems: technologies that form the foundation of many popular

applications and that have to meet scalability, performance, and reliability require‐

ments in production every day.

We will dig into the internals of those systems, tease apart their key algorithms, dis‐

cuss their principles and the trade-offs they have to make. On this journey, we will try

to find useful ways of thinking about data systems—not just how they work, but also

why they work that way, and what questions we need to ask.

After reading this book, you will be in a great position to decide which kind of tech‐

nology is appropriate for which purpose, and understand how tools can be combined

to form the foundation of a good application architecture. You won’t be ready to

build your own database storage engine from scratch, but fortunately that is rarely

necessary. You will, however, develop a good intuition for what your systems are

doing under the hood so that you can reason about their behavior, make good design

decisions, and track down any problems that may arise.

Who Should Read This Book?

If you develop applications that have some kind of server/backend for storing or pro‐

cessing data, and your applications use the internet (e.g., web applications, mobile

apps, or internet-connected sensors), then this book is for you.

xiv | Preface

This book is for software engineers, software architects, and technical managers who

love to code. It is especially relevant if you need to make decisions about the architec‐

ture of the systems you work on—for example, if you need to choose tools for solving

a given problem and figure out how best to apply them. But even if you have no

choice over your tools, this book will help you better understand their strengths and

weaknesses.

You should have some experience building web-based applications or network serv‐

ices, and you should be familiar with relational databases and SQL. Any non-

relational databases and other data-related tools you know are a bonus, but not

required. A general understanding of common network protocols like TCP and

HTTP is helpful. Your choice of programming language or framework makes no dif‐

ference for this book.

If any of the following are true for you, you’ll find this book valuable:

• You want to learn how to make data systems scalable, for example, to support

web or mobile apps with millions of users.

• You need to make applications highly available (minimizing downtime) and

operationally robust.

• You are looking for ways of making systems easier to maintain in the long run,

even as they grow and as requirements and technologies change.

•

You have a natural curiosity for the way things work and want to know what

goes on inside major websites and online services. This book breaks down the

internals of various databases and data processing systems, and it’s great fun to

explore the bright thinking that went into their design.

Sometimes, when discussing scalable data systems, people make comments along the

lines of, “You’re not Google or Amazon. Stop worrying about scale and just use a

relational database.” There is truth in that statement: building for scale that you don’t

need is wasted effort and may lock you into an inflexible design. In effect, it is a form

of premature optimization. However, it’s also important to choose the right tool for

the job, and different technologies each have their own strengths and weaknesses. As

we shall see, relational databases are important but not the final word on dealing with

data.

Scope of This Book

This book does not attempt to give detailed instructions on how to install or use spe‐

cific software packages or APIs, since there is already plenty of documentation for

those things. Instead we discuss the various principles and trade-offs that are funda‐

mental to data systems, and we explore the different design decisions taken by differ‐

ent products.

Preface | xv

In the ebook editions we have included links to the full text of online resources. All

links were verified at the time of publication, but unfortunately links tend to break

frequently due to the nature of the web. If you come across a broken link, or if you

are reading a print copy of this book, you can look up references using a search

engine. For academic papers, you can search for the title in Google Scholar to find

open-access PDF files. Alternatively, you can find all of the references at https://

github.com/ept/ddia-references, where we maintain up-to-date links.

We look primarily at the architecture of data systems and the ways they are integrated

into data-intensive applications. This book doesn’t have space to cover deployment,

operations, security, management, and other areas—those are complex and impor‐

tant topics, and we wouldn’t do them justice by making them superficial side notes in

this book. They deserve books of their own.

Many of the technologies described in this book fall within the realm of the Big Data

buzzword. However, the term “Big Data” is so overused and underdefined that it is

not useful in a serious engineering discussion. This book uses less ambiguous terms,

such as single-node versus distributed systems, or online/interactive versus offline/

batch processing systems.

This book has a bias toward free and open source software (FOSS), because reading,

modifying, and executing source code is a great way to understand how something

works in detail. Open platforms also reduce the risk of vendor lock-in. However,

where appropriate, we also discuss proprietary software (closed-source software, soft‐

ware as a service, or companies’ in-house software that is only described in literature

but not released publicly).

Outline of This Book

This book is arranged into three parts:

1. In Part I, we discuss the fundamental ideas that underpin the design of data-

intensive applications. We start in Chapter 1 by discussing what we’re actually

trying to achieve: reliability, scalability, and maintainability; how we need to

think about them; and how we can achieve them. In Chapter 2 we compare sev‐

eral different data models and query languages, and see how they are appropriate

to different situations. In Chapter 3 we talk about storage engines: how databases

arrange data on disk so that we can find it again efficiently. Chapter 4 turns to

formats for data encoding (serialization) and evolution of schemas over time.

In Part II, we move from data stored on one machine to data that is distributed

across multiple machines. This is often necessary for scalability, but brings with

it a variety of unique challenges. We first discuss replication (Chapter 5), parti‐

tioning/sharding (Chapter 6), and transactions (Chapter 7). We then go into

xvi | Preface

more detail on the problems with distributed systems (Chapter 8) and what it

means to achieve consistency and consensus in a distributed system (Chapter 9).

3. In Part III, we discuss systems that derive some datasets from other datasets.

Derived data often occurs in heterogeneous systems: when there is no one data‐

base that can do everything well, applications need to integrate several different

databases, caches, indexes, and so on. In Chapter 10 we start with a batch pro‐

cessing approach to derived data, and we build upon it with stream processing in

Chapter 11. Finally, in Chapter 12 we put everything together and discuss

approaches for building reliable, scalable, and maintainable applications in the

future.

References and Further Reading

Most of what we discuss in this book has already been said elsewhere in some form or

another—in conference presentations, research papers, blog posts, code, bug trackers,

mailing lists, and engineering folklore. This book summarizes the most important

ideas from many different sources, and it includes pointers to the original literature

throughout the text. The references at the end of each chapter are a great resource if

you want to explore an area in more depth, and most of them are freely available

online.

O’Reilly Safari

Safari (formerly Safari Books Online) is a membership-based

training and reference platform for enterprise, government,

educators, and individuals.

Members have access to thousands of books, training videos, Learning Paths, interac‐

tive tutorials, and curated playlists from over 250 publishers, including O’Reilly

Media, Harvard Business Review, Prentice Hall Professional, Addison-Wesley Pro‐

fessional, Microsoft Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco

Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt,

Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett,

and Course Technology, among others.

For more information, please visit http://oreilly.com/safari.

Preface | xvii

How to Contact Us

Please address comments and questions concerning this book to the publisher:

O’Reilly Media, Inc.

1005 Gravenstein Highway North

Sebastopol, CA 95472

800-998-9938 (in the United States or Canada)

707-829-0515 (international or local)

707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional

information. You can access this page at http://bit.ly/designing-data-intensive-apps.

To comment or ask technical questions about this book, send email to bookques‐

tions@oreilly.com.

For more information about our books, courses, conferences, and news, see our web‐

site at http://www.oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments

This book is an amalgamation and systematization of a large number of other peo‐

ple’s ideas and knowledge, combining experience from both academic research and

industrial practice. In computing we tend to be attracted to things that are new and

shiny, but I think we have a huge amount to learn from things that have been done

before. This book has over 800 references to articles, blog posts, talks, documenta‐

tion, and more, and they have been an invaluable learning resource for me. I am very

grateful to the authors of this material for sharing their knowledge.

I have also learned a lot from personal conversations, thanks to a large number of

people who have taken the time to discuss ideas or patiently explain things to me. In

particular, I would like to thank Joe Adler, Ross Anderson, Peter Bailis, Márton

Balassi, Alastair Beresford, Mark Callaghan, Mat Clayton, Patrick Collison, Sean

Cribbs, Shirshanka Das, Niklas Ekström, Stephan Ewen, Alan Fekete, Gyula Fóra,

Camille Fournier, Andres Freund, John Garbutt, Seth Gilbert, Tom Haggett, Pat Hel‐

land, Joe Hellerstein, Jakob Homan, Heidi Howard, John Hugg, Julian Hyde, Conrad

Irwin, Evan Jones, Flavio Junqueira, Jessica Kerr, Kyle Kingsbury, Jay Kreps, Carl

Lerche, Nicolas Liochon, Steve Loughran, Lee Mallabone, Nathan Marz, Caitie

xviii | Preface

剩余612页未读，继续阅读

sunmoonkkk

粉丝: 2
资源: 5

数据密集应用设计：构建可靠、可扩展与维护的系统全解析

Designing.Data-Intensive.Applications 设计数据密集型应用（html中文版全网首发）

Designing.Data-Intensive.Applications

Designing.Data-Intensive.Applications-2017

Designing-Data-Intensive-Applications 英文高清完整下载

Designing-Data-Intensive-Applications

designing-data-Intensive-applications:以数据为中心的应用设计研究

java版飞机大战源码-designing-data-intensive-applications:设计数据密集型应用程序

designing-data-intensive-applications-notes:有关设计数据密集型应用程序的说明（进行中）

Designing Data-Intensive Applications

卡通风格化魔法术技能粒子特效 ：Toon Projectiles 2 1.0

最新资源

卡通风格化魔法术技能粒子特效：Toon Projectiles 2 1.0