Bigtable：大规模结构化数据的分布式存储解决方案
需积分: 14 35 浏览量
更新于2024-09-12
收藏 216KB PDF 举报
Bigtable是由Google设计的一款分布式存储系统，专为结构化数据管理而生，旨在应对大规模数据处理的需求，能够跨越数千台普通服务器存储 petabytes 的数据。它在Google内部被广泛应用，支持如网页索引、Google Earth和Google Finance等项目，这些应用对数据规模和延迟需求差异巨大，从URL到网页再到卫星图像，以及后端批量处理到实时数据服务，Bigtable都能提供灵活且高性能的解决方案。
Bigtable的核心概念在于其分布式架构，通过将大量数据分散存储在多个节点上，实现了数据的横向扩展。这种设计允许系统在保持高可用性和容错性的同时，处理海量的数据量。它的数据模型非常灵活，客户端可以动态控制数据布局和格式，这使得它能够适应各种不同的应用场景。
Bigtable的设计包括以下几个关键组件：
1. **表格模型**（Table Model）：Bigtable的表格结构类似于传统的二维表格，但具有可伸缩性。每个表格由行键（Row Key）、列族（Column Family）和列（Columns）组成，允许灵活的数据组织和查询。
2. **分布式存储**：Bigtable利用分布式哈希表技术，将数据分布在整个集群中的多个节点上，通过一致性哈希算法确保数据的一致性。每个节点负责一部分数据，即使有节点故障，也能通过负载均衡实现数据的无缝接管。
3. **分片和复制**：为了处理不同数据大小和访问频率的差异，Bigtable会将数据划分为多个片段（Shards），并通过多副本策略提高数据可靠性。用户可以根据需求调整数据的冗余级别。
4. **列式存储**：Bigtable采用列式存储方式，即数据按照列族和列进行组织，而不是像关系数据库那样按行。这种设计有助于减少数据扫描的I/O开销，对于大量数据的实时查询效率较高。
5. **分布式索引**：为了加速数据查找，Bigtable维护了一个全局的二级索引，使得客户端可以高效地定位到数据所在的节点，即使数据在分布式存储中分散。
6. **一致性与性能**：Bigtable采用强一致性模型，保证了在同一时刻，所有节点看到的数据都是最新的。同时，它引入了预读和缓存策略来优化性能，确保低延迟的响应时间。
7. **可扩展性**：Bigtable的设计允许平滑地增加或减少服务器，无需大规模的数据迁移。这使得系统能够随着业务的增长持续适应。
Bigtable的成功在于其灵活的数据模型、高效的分布式存储和处理机制，以及针对大规模数据处理场景的精心设计。作为一款强大的分布式存储系统，Bigtable已经成为Google众多产品背后的关键基础设施，并为其他企业提供了处理结构化数据的高效解决方案。
// Open the table
Table *T = OpenOrDie("/bigtable/web/webtable");
// Write a new anchor and delete an old anchor
RowMutation r1(T, "com.cnn.www");
r1.Set("anchor:www.c-span.org", "CNN");
r1.Delete("anchor:www.abc.com");
Operation op;
Apply(&op, &r1);
Figure 2: Writing to Bigtable.
applications. Applications that need to avoid collisions
must generate unique timestamps themselves. Different
versions of a cell are stored in decreasing timestamp or-
der, so that the most recent versions can be read first.
To make the management of versioned data less oner-
ous, we support two per-column-family settings that tell
Bigtable to garbage-collect cell versions automatically.
The client can specify either that only the last n versions
of a cell be kept, or that only new-enough versions be
kept (e.g., only keep values that were written in the last
seven days).
In our Webtable example, we set the timestamps of
the crawled pages stored in the contents: column to
the times at which these page versions were actually
crawled. The garbage-collection mechanism described
above lets us keep only the most recent three versions of
every page.
3 API
The Bigtable API provides functions for creating and
deleting tables and column families. It also provides
functions for changing cluster, table, and column family
metadata, such as access control rights.
Client applications can write or delete values in
Bigtable, look up values from individual rows, or iter-
ate over a subset of the data in a table. Figure 2 shows
C++ code that uses a RowMutation abstraction to per-
form a series of updates. (Irrelevant details were elided
to keep the example short.) The call to Apply performs
an atomic mutation to the Webtable: it adds one anchor
to www.cnn.com and deletes a different anchor.
Figure 3 shows C++ code that uses a Scanner ab-
straction to iterate over all anchors in a particular row.
Clients can iterate over multiple column families, and
there are several mechanisms for limiting the rows,
columns, and timestamps produced by a scan. For ex-
ample, we could restrict the scan above to only produce
anchors whose columns match the regular expression
anchor:*.cnn.com, or to only produce anchors whose
timestamps fall within ten days of the current time.
Scanner scanner(T);
ScanStream *stream;
stream = scanner.FetchColumnFamily("anchor");
stream->SetReturnAllVersions();
scanner.Lookup("com.cnn.www");
for (; !stream->Done(); stream->Next()) {
printf("%s %s %lld %s\n",
scanner.RowName(),
stream->ColumnName(),
stream->MicroTimestamp(),
stream->Value());
}
Figure 3: Reading from Bigtable.
Bigtable supports several other features that allow the
user to manipulate data in more complex ways. First,
Bigtable supports single-row transactions, which can be
used to perform atomic read-modify-write sequences on
data stored under a single row key. Bigtable does not cur-
rently support general transactions across row keys, al-
though it provides an interface for batching writes across
row keys at the clients. Second, Bigtable allows cells
to be used as integer counters. Finally, Bigtable sup-
ports the execution of client-supplied scripts in the ad-
dress spaces of the servers. The scripts are written in a
language developed at Google for processing data called
Sawzall [28]. At the moment, our Sawzall-based API
does not allow client scripts to write back into Bigtable,
but it does allow various forms of data transformation,
filtering based on arbitrary expressions, and summariza-
tion via a variety of operators.
Bigtable can be used with MapReduce [12], a frame-
work for running large-scale parallel computations de-
veloped at Google. We have written a set of wrappers
that allow a Bigtable to be used both as an input source
and as an output target for MapReduce jobs.
4 Building Blocks
Bigtable is built on several other pieces of Google in-
frastructure. Bigtable uses the distributed Google File
System (GFS) [17] to store log and data files. A Bigtable
cluster typically operates in a shared pool of machines
that run a wide variety of other distributed applications,
and Bigtable processes often share the same machines
with processes from other applications. Bigtable de-
pends on a cluster management system for scheduling
jobs, managing resources on shared machines, dealing
with machine failures, and monitoring machine status.
The Google SSTable file format is used internally to
store Bigtable data. An SSTable provides a persistent,
ordered immutable map from keys to values, where both
keys and values are arbitrary byte strings. Operations are
provided to look up the value associated with a specified
To appear in OSDI 2006 3
剩余13页未读，继续阅读
218 浏览量
2014-01-04 上传
2016-11-25 上传
2013-01-05 上传
2014-02-19 上传
2011-05-09 上传
2017-12-11 上传
点击了解资源详情
点击了解资源详情
fhrihann21
- 粉丝: 64
- 资源: 10
我的内容管理
展开
最新资源
- Java集合ArrayList实现字符串管理及效果展示
- 实现2D3D相机拾取射线的关键技术
- LiveLy-公寓管理门户：创新体验与技术实现
- 易语言打造的快捷禁止程序运行小工具
- Microgateway核心：实现配置和插件的主端口转发
- 掌握Java基本操作：增删查改入门代码详解
- Apache Tomcat 7.0.109 Windows版下载指南
- Qt实现文件系统浏览器界面设计与功能开发
- ReactJS新手实验：搭建与运行教程
- 探索生成艺术：几个月创意Processing实验
- Django框架下Cisco IOx平台实战开发案例源码解析
- 在Linux环境下配置Java版VTK开发环境
- 29街网上城市公司网站系统v1.0：企业建站全面解决方案
- WordPress CMB2插件的Suggest字段类型使用教程
- TCP协议实现的Java桌面聊天客户端应用
- ANR-WatchDog: 检测Android应用无响应并报告异常
