探索HBase：权威指南 - CSDN文库

5星 · 超过95%的资源需积分: 9 36 浏览量更新于2024-07-30 收藏 5.56MB PDF 举报

"HBase The Definitive Guide.pdf" 《HBase权威指南》是Lars George撰写的一本关于HBase的详细教程，旨在帮助读者深入理解并掌握这个基于Hadoop的分布式列式数据库系统。该书由O'Reilly Media, Inc.出版，版权归属Lars George所有，首次发行于2011年。 HBase是一种建立在Hadoop之上的开源、非关系型（NoSQL）数据库，特别适合处理和存储海量结构化数据。它提供高并发读写能力，且具备强一致性和高可用性。HBase的设计理念是能够处理PB级别的数据，并且在大规模集群上运行。书中涵盖了HBase的核心概念和组件，包括： 1. **分布式架构**：介绍HBase如何利用Hadoop的HDFS（Hadoop Distributed File System）作为底层存储，并利用ZooKeeper进行元数据管理和协调。 2. **表模型**：解释HBase的行、列族、列和时间戳的概念，以及如何设计高效的表结构以适应查询模式。 3. **数据模型**：阐述HBase的键值对模型，包括RowKey（行键）的设计策略，它是HBase查询性能的关键因素。 4. **读写操作**：详细讨论Get、Put、Scan等操作，以及如何优化读写性能，包括布隆过滤器和预读取（BlockCache）机制。 5. **Region管理**：介绍Region的概念，以及RegionServer如何动态调整和分片，以保持负载均衡。 6. **数据一致性**：讨论HBase的一致性模型，包括单行事务和WAL（Write-Ahead Log）日志的角色。 7. **MapReduce与HBase**：讲解如何使用MapReduce与HBase交互，进行大数据分析和批量处理。 8. **监控与故障排查**：提供监控HBase集群的方法，包括指标收集和报警设置，以及如何应对常见的故障和问题。 9. **扩展与优化**：涵盖HBase的扩展性，如Sharding和Compaction，以及如何通过调优提高整体性能。 10. **实际案例**：通过真实案例展示HBase在不同场景下的应用，例如搜索引擎索引、日志分析和实时推荐系统。这本书不仅适合Hadoop和大数据领域的初学者，也对有经验的开发者提供了深入的洞察和实用的技巧。通过阅读《HBase权威指南》，读者可以全面了解HBase的工作原理，并具备实施和管理大型HBase集群的能力。

Foreword

The HBase story begins in 2006, when the San Francisco-based startup Powerset was

trying to build a natural language search engine for the Web. Their indexing pipeline

was an involved multistep process that produced an index about two orders of mag-

nitude larger, on average, than your standard term-based index. The datastore that

they’d built on top of the then nascent Amazon Web Services to hold the index inter-

mediaries and the webcrawl was buckling under the load (Ring. Ring. “Hello! This is

AWS. Whatever you are running, please turn it off!”). They were looking for an alter-

native. The Google BigTable paper

*

had just been published.

Chad Walters, Powerset’s head of engineering at the time, reflects back on the

experience as follows:

Building an open source system to run on top of Hadoop’s Distributed Filesystem (HDFS)

in much the same way that BigTable ran on top of the Google File System seemed like a

good approach because: 1) it was a proven scalable architecture; 2) we could leverage

existing work on Hadoop’s HDFS; and 3) we could both contribute to and get additional

leverage from the growing Hadoop ecosystem.

After the publication of the Google BigTable paper, there were on-again, off-again dis-

cussions around what a BigTable-like system on top of Hadoop might look. Then, in

early 2007, out of the blue, Mike Cafarela dropped a tarball of thirty odd Java files into

the Hadoop issue tracker: “I’ve written some code for HBase, a BigTable-like file store.

It’s not perfect, but it’s ready for other people to play with and examine.” Mike had

been working with Doug Cutting on Nutch, an open source search engine. He’d done

similar drive-by code dumps there to add features such as a Google File System clone

so the Nutch indexing process was not bounded by the amount of disk you attach to

a single machine. (This Nutch distributed filesystem would later grow up to be HDFS.)

Jim Kellerman of Powerset took Mike’s dump and started filling in the gaps, adding

tests and getting it into shape so that it could be committed as part of Hadoop. The

first commit of the HBase code was made by Doug Cutting on April 3, 2007, under

the contrib subdirectory. The first HBase “working” release was bundled as part of

Hadoop 0.15.0 in October 2007.

* “BigTable: A Distributed Storage System for Structured Data” by Fay Chang et al.

xv

Not long after, Lars, the author of the book you are now reading, showed up on the

#hbase IRC channel. He had a big-data problem of his own, and was game to try HBase.

After some back and forth, Lars became one of the first users to run HBase in production

outside of the Powerset home base. Through many ups and downs, Lars stuck around.

I distinctly remember a directory listing Lars made for me a while back on his produc-

tion cluster at WorldLingo, where he was employed as CTO, sysadmin, and grunt. The

listing showed ten or so HBase releases from Hadoop 0.15.1 (November 2007) on up

through HBase 0.20, each of which he’d run on his 40-node cluster at one time or

another during production.

Of all those who have contributed to HBase over the years, it is poetic justice that Lars

is the one to write this book. Lars was always dogging HBase contributors that the

documentation needed to be better if we hoped to gain broader adoption. Everyone

agreed, nodded their heads in ascent, amen’d, and went back to coding. So Lars started

writing critical how-tos and architectural descriptions inbetween jobs and his intra-

European travels as unofficial HBase European ambassador. His Lineland blogs on

HBase gave the best description, outside of the source, of how HBase worked, and at

a few critical junctures, carried the community across awkward transitions (e.g., an

important blog explained the labyrinthian HBase build during the brief period we

thought an Ivy-based build to be a “good idea”). His luscious diagrams were poached

by one and all wherever an HBase presentation was given.

HBase has seen some interesting times, including a period of sponsorship by Microsoft,

of all things. Powerset was acquired in July 2008, and after a couple of months during

which Powerset employees were disallowed from contributing while Microsoft’s legal

department vetted the HBase codebase to see if it impinged on SQLServer patents, we

were allowed to resume contributing (I was a Microsoft employee working near full

time on an Apache open source project). The times ahead look promising, too, whether

it’s the variety of contortions HBase is being put through at Facebook—as the under-

pinnings for their massive Facebook mail app or fielding millions of of hits a second on

their analytics clusters—or more deploys along the lines of Yahoo!’s 1k node HBase

cluster used to host their snapshot of Microsoft’s Bing crawl. Other developments in-

clude HBase running on filesystems other than Apache HDFS, such as MapR.

But plain to me though is that none of these developments would have been possible

were it not for the hard work put in by our awesome HBase community driven by a

core of HBase committers. Some members of the core have only been around a year or

so—Todd Lipcon, Gary Helmling, and Nicolas Spiegelberg—and we would be lost

without them, but a good portion have been there from close to project inception and

have shaped HBase into the (scalable) general datastore that it is today. These include

Jonathan Gray, who gambled his startup streamy.com on HBase; Andrew Purtell, who

built an HBase team at Trend Micro long before such a thing was fashionable; Ryan

Rawson, who got StumbleUpon—which became the main sponsor after HBase moved

on from Powerset/Microsoft—on board, and who had the sense to hire John-Daniel

Cryans, now a power contributor but just a bushy-tailed student at the time. And then

xvi | Foreword

剩余553页未读，继续阅读

webbean2

粉丝: 0
资源: 14

最新资源