探索Google分布式文件系统GFS：突破传统设计与应用实践

5星 · 超过95%的资源需积分: 32 115 浏览量更新于2024-07-24 2 收藏 257KB PDF 举报

"《Google三篇论文：GFS英文版》是一份深入探讨Google分布式文件系统（Google File System，GFS）的重要资料。该论文由Sanjay Ghemawat、Howard Gobioff和Shun-Tak Leung三位Google工程师共同撰写，发表于Google内部，旨在分享他们在设计和实现GFS过程中的关键洞察。 GFS的出现是对传统文件系统设计理念的革新，它针对大规模、数据密集型应用的需求进行了特别设计。其核心目标在于提供在廉价商用硬件上运行时的高可用性和容错能力，同时确保对众多客户端的高性能访问。与早期的分布式文件系统相比，GFS的设计思路受到了Google自身业务工作负载和技术环境的深刻影响，尤其是那些与存储需求紧密相关的应用和服务。论文详细阐述了GFS的设计原则，包括如何通过分布式架构来分散数据存储，提高系统的扩展性；如何利用数据分片（Data Chunking）技术，使得单个文件被划分为多个小块分布在不同的服务器上，从而实现故障隔离和并行读写；以及如何通过副本策略保证数据一致性，即使在部分节点失效时也能保证服务的连续性。GFS的成功之处在于其能够在大型集群中提供数百TB的存储容量，跨越数千台机器，服务于Google的各项服务，如数据生成和处理，以及研究和开发活动，这些都依赖于大容量的数据集。此外，GFS还强调了与Hadoop等大数据处理框架的集成，这表明其在大数据时代的重要性。通过与Hadoop MapReduce的无缝协作，GFS不仅解决了存储问题，还优化了整个数据处理流程的性能。总结来说，《Google三篇论文：GFS英文版》揭示了Google在面对特定业务需求和技术挑战时，如何打破传统，创新分布式文件系统设计，从而推动了整个云计算和大数据领域的发展。这份文档对于理解现代分布式存储系统的设计理念和技术细节具有很高的参考价值。"

Legend:

Data messages

Control messages

Application

(file name, chunk index)

(chunk handle,

chunk locations)

GFS master

File namespace

/foo/bar

Instructions to chunkserver

Chunkserver state

GFS chunkserverGFS chunkserver

(chunk handle, byte range)

chunk data

chunk 2ef0

Linux file system Linux file system

GFS client

Figure 1: GFS Architecture

and replication decisions using global knowledge. However,

we must minimize its involvement in reads and writes so

that it does not become a bottleneck. Clients never read

and write ﬁle data through the master. Instead, a client asks

the master which chunkservers it should contact. It caches

this information for a limited time and interacts with the

chunkservers directly for many subsequent operations.

Let us explain the interactions for a simple read with refer-

ence to Figure 1. First, using the ﬁxed chunk size, the client

translates the ﬁle name and byte oﬀset speciﬁed by the ap-

plication into a chunk index within the ﬁle. Then, it sends

the master a request containing the ﬁle name and chunk

index. The master replies with the corresponding chunk

handle and locations of the replicas. The client caches this

information using the ﬁle name and chunk index as the key.

The client then sends a request to one of the replicas,

most likely the closest one. The request speciﬁes the chunk

handle and a byte range within that chunk. Further reads

of the same chunk require no more client-master interaction

until the cached information expires or the ﬁle is reopened.

In fact, the client typically asks for multiple chunks in the

same request and the master can also include the informa-

tion for chunks immediately following those requested. This

extra information sidesteps several future client-master in-

teractions at practically no extra cost.

2.5 Chunk Size

Chunk size is one of the key design parameters. We have

chosen 64 MB, which is much larger than typical ﬁle sys-

tem block sizes. Each chunk replica is stored as a plain

Linux ﬁle on a chunkserver and is extended only as needed.

Lazy space allocation avoids wasting space due to internal

fragmentation, perhaps the greatest objection against such

a large chunk size.

A large chunk size oﬀers several important advantages.

First, it reduces clients’ need to interact with the master

because reads and writes on the same chunk require only

one initial request to the master for chunk location informa-

tion. The reduction is especially signiﬁcant for our work-

loads because applications mostly read and write large ﬁles

sequentially. Even for small random reads, the client can

comfortably cache all the chunk location information for a

multi-TB working set. Second, since on a large chunk, a

client is more likely to perform many operations on a given

chunk, it can reduce network overhead by keeping a persis-

tent TCP connection to the chunkserver over an extended

period of time. Third, it reduces the size of the metadata

stored on the master. This allows us to keep the metadata

in memory, which in turn brings other advantages that we

will discuss in Section 2.6.1.

On the other hand, a large chunk size, even with lazy space

allocation, has its disadvantages. A small ﬁle consists of a

small number of chunks, perhaps just one. The chunkservers

storing those chunks may become hot spots if many clients

are accessing the same ﬁle. In practice, hot spots have not

been a major issue because our applications mostly read

large multi-chunk ﬁles sequentially.

However, hot spots did develop when GFS was ﬁrst used

by a batch-queue system: an executable was written to GFS

as a single-chunk ﬁle and then started on hundreds of ma-

chines at the same time. The few chunkservers storing this

executable were overloaded by hundreds of simultaneous re-

quests. We ﬁxed this problem by storing such executables

with a higher replication factor and by making the batch-

queue system stagger application start times. A potential

long-term solution is to allow clients to read data from other

clients in such situations.

2.6 Metadata

The master stores three major types of metadata: the ﬁle

and chunk namespaces, the mapping from ﬁles to chunks,

and the locations of each chunk’s replicas. All metadata is

kept in the master’s memory. The ﬁrst two types (names-

paces and ﬁle-to-chunk mapping) are also kept persistent by

logging mutations to an operation log stored on the mas-

ter’s local disk and replicated on remote machines. Using

a log allows us to update the master state simply, reliably,

and without risking inconsistencies in the event of a master

crash. The master does not store chunk location informa-

tion persistently. Instead, it asks each chunkserver about its

chunks at master startup and whenever a chunkserver joins

the cluster.

2.6.1 In-Memory Data Structures

Since metadata is stored in memory, master operations are

fast. Furthermore, it is easy and eﬃcient for the master to

periodically scan through its entire state in the background.

This periodic scanning is used to implement chunk garbage

collection, re-replication in the presence of chunkserver fail-

ures, and chunk migration to balance load and disk space

剩余14页未读，继续阅读

bogehahaha

粉丝: 1
资源: 4

探索Google分布式文件系统GFS：突破传统设计与应用实践

Google三篇论文之GFS（中文版）

GFS论文中英文版.rar

Google大数据经典论文（GFS/BigTable/MapReduce）

Google 三篇论文 英文版 中文版

谷歌GFS+Mapreduce+Bigtable三大论文中英文+PDF+WORD版本

Hadoop 权威指南（英文第3版）.pdf

谷歌3篇大数据论文中英文

gfs+BigTable+map reduce 三合一.zip

google大数据论文（中英文）

Google三大论文

最新资源

Google 三篇论文英文版中文版