突破传统：Google文件系统GFS——云计算与大数据基石

需积分: 10 25 浏览量更新于2024-07-29 收藏 269KB PDF 举报

《谷歌文件系统》（Google File System, GFS）是Google的三大杰出技术成就之一，它为现代云计算和大规模数据处理提供了关键的基础。本文由Sanjay Ghemawat、Howard Gobioff和Shun-Tak Leung三位作者共同撰写，他们在Google内部设计并实现了这个分布式文件系统，旨在满足大型分布式数据密集型应用的需求。 GFS的设计理念主要源于对Google内部工作负载的深入理解，以及对当前和预期技术环境的洞察。它挑战了早期文件系统的一些假设，特别是在面对大规模数据存储和处理时。相比于传统的文件系统设计，GFS更侧重于以下几点： 1. **可扩展性与容错**：GFS能够在低成本的商业硬件上运行，并能提供高可用性和容错能力，确保在单个节点故障时，数据仍能保持完整性。 2. **性能优化**：为了服务众多客户端，GFS特别关注整体性能的提升，包括数据读写速度和吞吐量，这对于大数据集的处理至关重要。 3. **适应应用需求**：GFS针对Google服务的数据生成、处理和研究开发项目设计，能够支持海量数据存储，一个集群可能跨越数千台机器，每台机器包含数千个磁盘，达到数千TB的存储容量。 4. **分布架构**：GFS采用分布式架构，数据被分割成多个小块（通常为64MB），分散存储在不同的服务器上，这使得数据访问更加高效，减少了网络延迟。 5. **动态扩展**：随着业务增长，GFS可以轻松地添加或移除节点，以适应变化的数据存储需求，体现了其灵活和弹性。 6. **数据一致性**：尽管是分布式系统，GFS通过复制策略和数据校验机制确保数据的一致性，即使在网络不稳定或个别节点故障时，也能维持数据的正确性。 GFS的成功应用不仅限于Google内部，它也成为了云计算和大数据领域的重要参考。许多后续的存储和计算平台都受到了GFS设计理念的启发，它对于现代IT基础设施的演进产生了深远影响。随着技术的进步，GFS可能已经进行了更新和改进，但其核心思想——为大规模、高性能和容错的分布式存储设计——依然在今天的云计算世界中发挥着关键作用。

Legend:

Data messages

Control messages

Application

(file name, chunk index)

(chunk handle,

chunk locations)

GFS master

File namespace

/foo/bar

Instructions to chunkserver

Chunkserver state

GFS chunkserverGFS chunkserver

(chunk handle, byte range)

chunk data

chunk 2ef0

Linux file system Linux file system

GFS client

Figure 1: GFS Architecture

and replication decisions using global knowledge. However,

we must minimize its involvement in reads and writes so

that it does not become a bottleneck. Clients never read

and write ﬁle data through the master. Instead, a client asks

the master which chunkservers it should contact. It caches

this information for a limited time and interacts with the

chunkservers directly for many subsequent operations.

Let us explain the interactions for a simple read with refer-

ence to Figure 1. First, using the ﬁxed chunk size, the client

translates the ﬁle name and byte oﬀset speciﬁed by the ap-

plication into a chunk index within the ﬁle. Then, it sends

the master a request containing the ﬁle name and chunk

index. The master replies with the corresponding chunk

handle and locations of the replicas. The client caches this

information using the ﬁle name and chunk index as the key.

The client then sends a request to one of the replicas,

most likely the closest one. The request speciﬁes the chunk

handle and a byte range within that chunk. Further reads

of the same chunk require no more client-master interaction

until the cached information expires or the ﬁle is reopened.

In fact, the client typically asks for multiple chunks in the

same request and the master can also include the informa-

tion for chunks immediately following those requested. This

extra information sidesteps several future client-master in-

teractions at practically no extra cost.

2.5 Chunk Size

Chunk size is one of the key design parameters. We have

chosen 64 MB, which is much larger than typical ﬁle sys-

tem block sizes. Each chunk replica is stored as a plain

Linux ﬁle on a chunkserver and is extended only as needed.

Lazy space allocation avoids wasting space due to internal

fragmentation, perhaps the greatest objection against such

a large chunk size.

A large chunk size oﬀers several important advantages.

First, it reduces clients’ need to interact with the master

because reads and writes on the same chunk require only

one initial request to the master for chunk location informa-

tion. The reduction is especially signiﬁcant for our work-

loads because applications mostly read and write large ﬁles

sequentially. Even for small random reads, the client can

comfortably cache all the chunk location information for a

multi-TB working set. Second, since on a large chunk, a

client is more likely to perform many operations on a given

chunk, it can reduce network overhead by keeping a persis-

tent TCP connection to the chunkserver over an extended

period of time. Third, it reduces the size of the metadata

stored on the master. This allows us to keep the metadata

in memory, which in turn brings other advantages that we

will discuss in Section 2.6.1.

On the other hand, a large chunk size, even with lazy space

allocation, has its disadvantages. A small ﬁle consists of a

small number of chunks, perhaps just one. The chunkservers

storing those chunks may become hot spots if many clients

are accessing the same ﬁle. In practice, hot spots have not

been a major issue because our applications mostly read

large multi-chunk ﬁles sequentially.

However, hot spots did develop when GFS was ﬁrst used

by a batch-queue system: an executable was written to GFS

as a single-chunk ﬁle and then started on hundreds of ma-

chines at the same time. The few chunkservers storing this

executable were overloaded by hundreds of simultaneous re-

quests. We ﬁxed this problem by storing such executables

with a higher replication factor and by making the batch-

queue system stagger application start times. A potential

long-term solution is to allow clients to read data from other

clients in such situations.

2.6 Metadata

The master stores three major types of metadata: the ﬁle

and chunk namespaces, the mapping from ﬁles to chunks,

and the locations of each chunk’s replicas. All metadata is

kept in the master’s memory. The ﬁrst two types (names-

paces and ﬁle-to-chunk mapping) are also kept persistent by

logging mutations to an operation log stored on the mas-

ter’s local disk and replicated on remote machines. Using

a log allows us to update the master state simply, reliably,

and without risking inconsistencies in the event of a master

crash. The master does not store chunk location informa-

tion persistently. Instead, it asks each chunkserver about its

chunks at master startup and whenever a chunkserver joins

the cluster.

2.6.1 In-Memory Data Structures

Since metadata is stored in memory, master operations are

fast. Furthermore, it is easy and eﬃcient for the master to

periodically scan through its entire state in the background.

This periodic scanning is used to implement chunk garbage

collection, re-replication in the presence of chunkserver fail-

ures, and chunk migration to balance load and disk space

剩余14页未读，继续阅读

daiwkiii

粉丝: 0
资源: 3

突破传统：Google文件系统GFS——云计算与大数据基石

【SCI2区】基于VMD-凌日优化算法TSOA-LSTM光伏预测Matlab实现.rar

天池大数据竞赛：降雨预测.zip

Axiom-0.7.5-py2-none-any.whl.zip

Proteus是英国Lab Center Electronics公司出版的EDA工具软件，是一款功能强大的电子电路仿真和物理特性

服务网格(Service Mesh)的istio产品实战教程资料-上传-课件

基于ssm的网上电子书店设计与实现.docx

百度-西交大·大数据竞赛2018复赛——商家招牌智能检测.zip

【SCI2区】黑猩猩优化算法Chimp-CNN-GRU-Attention用电需求预测Matlab实现.rar

基于ssm的课堂教学效果实时评价系统设计与实现.docx

景区智慧旅游系统建设解决方案PPT(46页).pptx

最新资源