企业级大数据平台架构指南：迈向Hadoop规模化

需积分: 9 2 浏览量更新于2024-07-17 收藏 20.77MB PDF 举报

《构建现代数据平台：企业级Hadoop规模化实践指南》是一本由Jan Kunigk、Ian Buss、Paul Wilkinson和Lars George合著的专业书籍，针对现代IT行业中的数据平台设计和架构提供了深入的洞察。该书专注于在大规模环境中实现企业级Hadoop的架构，这是一项关键的技术领域，尤其是在大数据处理和分析的时代。作者们结合自身丰富的经验，探讨了如何在企业级环境中有效部署、优化和管理Hadoop集群，涵盖了从基础设施构建、数据集成、存储策略到安全性和可扩展性的各个方面。书中不仅介绍了技术原理，还提供了实用的架构设计原则和最佳实践，帮助读者理解和应对在实际项目中可能遇到的各种挑战。读者可以预期本书会深入剖析Hadoop生态系统，包括HDFS（Hadoop分布式文件系统）、YARN（Yet Another Resource Negotiator）以及MapReduce、Hive、Pig等组件的工作原理。此外，书中还会涉及如何利用Hadoop与NoSQL数据库、Spark等其他技术进行协同，以实现高效的数据处理和分析。对于那些负责IT架构或数据科学团队的企业和技术领导者来说，这本书是一本宝贵的参考资料，它可以帮助他们构建更加稳健、可扩展和高可用的数据平台，以支持企业的决策制定和业务增长。通过阅读这本书，读者可以学习到如何在满足高性能需求的同时，确保数据安全、合规性和成本效益。版权方面，本书受到Jan Kunigk、Lars George、Ian Buss和Paul Wilkinson的共同保护，所有权利保留。《构建现代数据平台》由O'Reilly Media Inc.出版，强调了在美国印刷的品质，并提供在线版本供读者访问。对于教育、商业或销售推广用途，读者可以通过O'Reilly官网获取更多信息，或联系其机构销售部门。编辑团队包括Nicole Tache和Michele Cronin，生产编辑则由Nicholas Adams和Kristen Brown负责，而Copyeditor Shannon Wright和Proofreader Rachel Head确保了内容的准确性和专业性。最后，Ellen Trou作为Indexer负责书目索引的制作，使读者能够更方便地查找所需的信息。《构建现代数据平台》是一本不可多得的实操指南，为企业级Hadoop架构师和数据平台工程师提供了深入且实用的参考，助力他们在大数据时代成功驾驭数据的力量。

associated services—that exists today. The thinking and design, though, are tied not

to specific projects but to the fundamental architecture that made Hadoop successful:

large-scale, distributed, shared-nothing software requires a new approach to opera‐

tions, to security, and to governance.

You will learn those techniques and those ideas here.

— Mike Olson

Founder and Chief Strategy Officer at Cloudera

Foreword | xv

Preface

If you’re reading this book, it will come as no surprise that we are in the middle of a

revolution in the way data is stored and processed in the enterprise. As anyone who

has been in IT for any length of time knows, the technologies and approaches behind

data processing and storage are always evolving. However, in the past 10 to 15 years,

the pace of change has been remarkable. We have moved from a world where almost

all enterprise data was processed and analyzed using variants of SQL and was con‐

tained in some form of relational database to one in which an enterprise’s data may

be found in a variety of so-called NoSQL storage engines. Each of these engines sacri‐

fices some constraint of the relational model to achieve superior performance and

scalability for a certain use case. The modern data landscape includes nonrelational

key-value stores, distributed filesystems, distributed columnar databases, log stores,

and document stores, in addition to traditional relational databases. The data in these

systems is exploited in a multitude of ways and is processed using distributed batch-

processing algorithms, stream processing, massively parallel processing query

engines, free-text searches, and machine learning pipelines.

There are many drivers for this transformation, but the predominant ones are:

Volume

The phrase big data has been used too much to retain much value, but the sheer

volume of data generated by today’s enterprises, especially those with a heavy

web presence—which is to say all enterprises—is staggering. The explosion of

data from edge computing and Internet of Things (IoT) devices will only add to

the volume. Although storing data in as granular a form as possible may not

seem immediately useful, this will become increasingly important in order to

derive new insights. Storage is cheap, and bad decisions that have lasting conse‐

quences are costly. Better to store in full fidelity with a modern data platform and

have the option to make a new decision later. Traditional architectures based on

relational databases and shared file storage are simply unable to store and pro‐

cess data at these scales. This has led directly to the development of new tools

Preface | xvii

and techniques in which computations are linearly scalable and distributed by

default.

Velocity

Gone are the days in which data for analytics would arrive in nice, neat daily

batches. Although this still happens for some datasets, increasingly data arrives

in a streaming fashion at high rates. The velocity of its generation demands a new

way of storing, processing, and serving it up.

Variety

New insights and new models feed off data—the more the better. Hitherto

untapped sources of data, perhaps in semi-structured or completely unstructured

forms, are increasingly in demand. All aspects of an enterprise’s operation are

relevant and potentially valuable sources of information to drive new insights

and, ultimately, revenue. It’s essential to have a single, unified platform with

technologies capable of storing and processing all these many and varied forms

of data.

Competition

The enterprises that will succeed in the data age are the ones building new busi‐

ness strategies and products and, crucially, making decisions based on the

insights gleaned from new data sources. To make the right data-driven decisions,

you need a solid data and computation platform. Such a platform needs to be

capable of embracing both on-premises and cloud deployments. It also needs to

scale to support traditional data analytics and to enable advances in your busi‐

ness from data science, machine learning, and artificial intelligence (AI).

Some Misconceptions

We have only just begun our exploration of Hadoop in the enterprise, but it is worth

dispelling some common misconceptions about data platforms and Hadoop early on:

Data in Hadoop is schemaless

Although it is true that many technologies in the Hadoop ecosystem have more

flexible notions of schemas and do not impose schemas as strictly as, say, a rela‐

tional database, it is a mistake to think that data stored in Hadoop clusters does

not need a defined schema. Applications using data stored in Hadoop still need

to understand the data they are querying, and there is always some sort of under‐

lying data model or structure, either implicit or explicit. What the Hadoop eco‐

system does offer is much more flexibility in the way data is structured and

queried. Instead of imposing a globally fixed schema on the data as it is ingested

and potentially dropping any fields that don’t match the schema, the data gets its

structure from the frameworks and applications using it. This concept is often

referred to as schema on read. You can store any type of data in its raw form and

xviii | Preface

then process, transform, and combine it with other sources into the best format

and structure for your use case. And if you get it wrong, you can always build a

new representation from the raw data.

One copy of the data

This is a very common mistake when thinking about modern data platforms.

Different use cases require different access patterns, and this often means storing

the same datasets in different ways using different storage engines. This is a logi‐

cal consequence of the various optimizations each storage engine provides. This

data duplication should be considered par for the course and embraced as a fun‐

damental aspect of the freedom of operating in the Hadoop ecosystem. Hadoop

platforms are designed to be horizontally scalable and to be orders of magnitude

cheaper (if your enterprise IT department has a sensible approach to procure‐

ment, that is) than the proprietary alternatives. But the money you save on stor‐

age is just one aspect—maybe not even the most important aspect—of moving to

a modern data platform. What it also brings you is a multitude of options for

processing and querying the data and for extracting new value through scalable

analytics and machine learning.

One huge cluster

In the initial excitement of moving to Hadoop, the notion of a single, all-

encompassing data lake arose, in which all data was stored in and all processing

and querying were performed on a single cluster, which consisted of potentially

many thousands of machines. Although Hadoop is certainly capable of scaling to

that number of servers, the variety of access patterns and modes of processing

data don’t necessarily mesh well in a single cluster. Colocating use cases that

require strict query completion time guarantees with other ad hoc, variable

workloads is likely to lead to an unsatisfactory experience. Multitenancy controls

do exist, but they can’t change the fact that a finite set of resources can’t satisfy all

requirements all the time. As a result, you should plan for multiple clusters serv‐

ing different use cases with similar processing patterns or service levels. Don’t go

too far the other way, though. Lots of small clusters can be just as bad as a “single

cluster to rule them all.” Clusters can and should be shared, but be prepared to

divide and conquer when necessary.

Some General Trends

The trends in industry are clear to see. Many, if not most, enterprises have already

embarked on their data-driven journeys and are making serious investments in hard‐

ware, software, and services. The big data market is projected to continue growing

apace, reaching somewhere in the region of $90 billion of annual revenue by 2025.

Related markets, such as deep learning and artificial intelligence, that are enabled by

data platforms are also set to see exponential growth over the next decade.

Preface | xix

剩余632页未读，继续阅读

caofeng891102

粉丝: 172
资源: 1260

企业级大数据平台架构指南：迈向Hadoop规模化

Architecting Modern Data Platforms(2019)

Architecting Modern Data Platforms.epub

Architecting Modern Data Platforms: A Guide to Enterprise Hadoop at Scale

Architecting Modern Web Applications

Architecting Modern iOS Games

Architecting Modern Apps,Part 2

Architecting Modern Java EE Applications epub

Architecting Modern Apps,Part 1

Architecting Modern Java EE Applications 无水印pdf

Architecting Modern Java EE Applications azw3

最新资源