大数据系统设计实战与最佳实践全览

需积分: 9 193 浏览量更新于2024-07-20 收藏 4.25MB PDF 举报

《大数据：系统设计与最佳实践》是一本由 Nathan Marz 和 James Warren 合著的专业书籍，旨在深入探讨大规模实时数据系统的原理和实用策略。该书是Manning出版社的作品，专为那些希望在大数据领域进行系统设计和优化的读者提供详尽的知识和经验分享。书中涵盖了大数据处理的核心概念，包括数据收集、存储、处理和分析的挑战，以及如何构建可扩展、高性能和实时响应的系统。作者强调了设计原则和最佳实践，这些对于在当今高度数据驱动的世界中保持竞争力至关重要。作者们凭借他们在业界的实际经验和深厚理论基础，为读者提供了实用的指导，帮助他们避免常见陷阱并实现高效的数据处理架构。读者可以预期在本书中找到的内容包括： 1. 系统设计基础：理解数据流和事件驱动架构，以及它们在处理海量数据时的优势。 2. 数据模型与存储：学习如何选择合适的数据模型（如键值对、文档、列族或图形数据库），以及如何有效地存储和索引数据以支持实时查询。 3. 分布式计算框架：介绍Apache Hadoop及其生态系统中的关键组件，如HDFS和MapReduce，以及如何设计分布式计算任务。 4. 实时处理：探索Kafka、Storm等实时流处理技术，以及如何构建低延迟的数据管道。 5. 性能优化：了解如何通过监控、调整和负载均衡来提高系统的响应速度和吞吐量。 6. 容错性和可靠性：讨论如何设计容错机制，确保系统在面对故障时仍能保持高可用性。 7. 安全和隐私：关注数据保护和隐私法规，如何在设计系统时融入安全考虑。《大数据：系统设计与最佳实践》不仅适合数据工程师、架构师，也适用于希望通过理解和应用最佳实践提升业务洞察力的决策者。这本书不仅是技术指南，也是实战手册，提供了宝贵的经验教训和案例研究，使读者能够更好地应对大数据时代的复杂挑战。通过阅读这本书，读者将掌握一套完整的工具和技术，以构建未来数据驱动的世界中的核心系统。

acknowledgments

This book would not have been possible without the help and support of so many

individuals around the world. I must start with my parents, who instilled in me from a

young age a love of learning and exploring the world around me. They always encour-

aged me in all my career pursuits.

Likewise, my brother Iorav encouraged my intellectual interests from a young age.

I still remember when he taught me Algebra while I was in elementary school. He was

the one to introduce me to programming for the first time—he taught me Visual

Basic as he was taking a class on it in high school. Those lessons sparked a passion for

programming that led to my career.

I am enormously grateful to Michael Montano and Christopher Golda, the found-

ers of BackType. From the moment they brought me on as their first employee, I was

given an extraordinary amount of freedom to make decisions. That freedom was

essential for me to explore and exploit the Lambda Architecture to its fullest. They

never questioned the value of open source and allowed me to open source our tech-

nology liberally. Getting deeply involved with open source has been one of the great

privileges of my life.

Many of my professors from my time as a student at Stanford deserve special

thanks. Tim Roughgarden is the best teacher I’ve ever had—he radically improved my

ability to rigorously analyze, deconstruct, and solve difficult problems. Taking as many

classes as possible with him was one of the best decisions of my life. I also give thanks

to Monica Lam for instilling within me an appreciation for the elegance of Datalog.

Many years later I married Datalog with MapReduce to produce my first significant

open source project, Cascalog.

Licensed to Mike He <capitain@gmail.com>

ACKNOWLEDGMENTS

xvi

Chris Wensel was the first one to show me that processing data at scale could be elegant

and performant. His Cascading library changed the way I looked at Big Data processing.

None of my work would have been possible without the pioneers of the Big Data field.

Special thanks to Jeffrey Dean and Sanjay Ghemawat for the original MapReduce paper,

Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati,

Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall and

Werner Vogels for the original Dynamo paper, and Michael Cafarella and Doug Cutting

for founding the Apache Hadoop project.

Rich Hickey has been one of my biggest inspirations during my programming

career. Clojure is the best language I have ever used, and I’ve become a better pro-

grammer having learned it. I appreciate its practicality and focus on simplicity. Rich’s

philosophy on state and complexity in programming has influenced me deeply.

When I started writing this book, I was not nearly the writer I am now. Renae Gre-

goire, one of my development editors at Manning, deserves special thanks for helping

me improve as a writer. She drilled into me the importance of using examples to lead

into general concepts, and she set off many light bulbs for me on how to effectively

structure technical writing. The skills she taught me apply not only to writing techni-

cal books, but to blogging, giving talks, and communication in general. For gaining an

important life skill, I am forever grateful.

This book would not be nearly of the same quality without the efforts of my co-

author James Warren. He did a phenomenal job absorbing the theoretical concepts

and finding even better ways to present the material. Much of the clarity of the book

comes from his great communication skills.

My publisher, Manning, was a pleasure to work with. They were patient with me

and understood that finding the right way to write on such a big topic takes time.

Through the whole process they were supportive and helpful, and they always gave me

the resources I needed to be successful. Thanks to Marjan Bace and Michael Stephens

for all the support, and to all the other staff for their help and guidance along the way.

I try to learn as much as possible about writing from studying other writers. Brad-

ford Cross, Clayton Christensen, Paul Graham, Carl Sagan, and Derek Sivers have

been particularly influential.

Finally, I can’t give enough thanks to the hundreds of people who reviewed, com-

mented, and gave feedback on our book as it was being written. That feedback led us

to revise, rewrite, and restructure numerous times until we found ways to present the

material effectively. Special thanks to Aaron Colcord, Aaron Crow, Alex Holmes, Arun

Jacob, Asif Jan, Ayon Sinha, Bill Graham, Charles Brophy, David Beckwith, Derrick

Burns, Douglas Duncan, Hugo Garza, Jason Courcoux, Jonathan Esterhazy, Karl

Kuntz, Kevin Martin, Leo Polovets, Mark Fisher, Massimo Ilario, Michael Fogus,

Michael G. Noll, Patrick Dennis, Pedro Ferrera Bertran, Philipp Janert, Rodrigo

Abreu, Rudy Bonefas, Sam Ritchie, Siva Kalagarla, Soren Macbeth, Timothy Chk-

lovski, Walid Farid, and Zhenhua Guo.

ATHAN MARZ

Licensed to Mike He <capitain@gmail.com>

ABOUT THIS BOOK

xix

concepts that hold true regardless of existing tools, while illustration chapters use

real-world tools to demonstrate the concepts. Don’t let the names fool you, though—

all chapters are highly example-driven.

Chapters 2 through 9 focus on the batch layer of the Lambda Architecture. Here you

will learn about modeling your master dataset, using batch processing to create arbitrary

views of your data, and the trade-offs between incremental and batch processing.

Chapters 10 and 11 focus on the serving layer, which provides low latency access to

the views produced by the batch layer. Here you will learn about specialized databases

that are only written to in bulk. You will discover that these databases are dramatically

simpler than traditional databases, giving them excellent performance, operational,

and robustness properties.

Chapters 12 through 17 focus on the speed layer, which compensates for the batch

layer’s high latency to provide up-to-date results for all queries. Here you will learn

about No

SQL databases, stream processing, and managing the complexities of incre-

mental computation.

Chapter 18 uses your new-found knowledge to review the Lambda Architecture

once more and fill in any remaining gaps. You’ll learn about incremental batch pro-

cessing, variants of the basic Lambda Architecture, and how to get the most out of

your resources.

Code downloads and conventions

The source code for the book can be found at https://github.com/Big-Data-Manning.

We have provided source code for the running example SuperWebAnalytics.com.

Much of the source code is shown in numbered listings. These listings are meant

to provide complete segments of code. Some listings are annotated to help highlight

or explain certain parts of the code. In other places throughout the text, code frag-

ments are used when necessary.

Courier

typeface

is used to denote code for Java. In

both the listings and fragments, we make use of a

bold

code

font

to help identify key

parts of the code that are being explained in the text.

Author Online

Purchase of Big Data includes free access to a private web forum run by Manning Pub-

lications where you can make comments about the book, ask technical questions, and

receive help from the authors and other users. To access the forum and subscribe to

it, point your web browser to www.manning.com/BigData. This Author Online (

AO)

page provides information on how to get on the forum once you’re registered, what

kind of help is available, and the rules of conduct on the forum.

Manning’s commitment to our readers is to provide a venue where a meaningful

dialog among individual readers and between readers and the authors can take place.

It’s not a commitment to any specific amount of participation on the part of the

authors, whose contribution to the

AO forum remains voluntary (and unpaid). We

suggest you try asking the authors some challenging questions, lest their interest stray!

Licensed to Mike He <capitain@gmail.com>

剩余329页未读，继续阅读

capitainhe

粉丝: 0
资源: 4

大数据系统设计实战与最佳实践全览

基于大数据的智能交通分析系统的设计与实现

大数据-算法-气体喷射器的设计及数值模.pdf

系统架构设计师教程 pan

深入理解kafka(核心设计与实践原理) pdf

2021下半年软件设计师

designing data-intensive applications pdf

成为高级程序员的详细计划书

考研软件工程需要学什么

系统运维工程师面试题和每天的工作内容还有需要学习什么

最新资源