Google File System：大规模分布式存储的创新设计与应用

需积分: 10 104 浏览量更新于2024-07-25 收藏 269KB PDF 举报

Google File System (GFS) 是谷歌的三驾马车之一，它是一个专为大规模分布式数据密集型应用设计的分布式文件系统。本文由 Sanjay Ghemawat、Howard Gobioff 和 Shun-Tak Leung 联合撰写，发表于 Google 内部，旨在解决当时在廉价商用硬件上运行时对高可用性和性能的需求。 GFS 的设计理念与早期的分布式文件系统有着显著的区别，它的设计受到了 Google 应用工作负载和技术环境的深刻影响。这些工作负载和环境要求对存储需求有极高的灵活性和可扩展性，包括处理海量数据和在数千台机器上分布存储。与传统的文件系统假设相比，GFS 更加注重在成本效益和性能之间找到平衡，同时保证了系统的故障容忍度。 GFS 成功地满足了Google的存储需求，不仅用于支撑其服务的数据生成和处理，还支持了需要大量数据集的研究和开发工作。截至报告撰写时，最大的集群已跨越数千台机器，提供了数百太字节的存储空间，显示了其在大规模分布式环境中的高效运作能力。 GFS 的核心特点包括： 1. **水平扩展**：通过增加节点数来扩大存储容量和处理能力，而非垂直扩展，这使得GFS能够轻松应对快速增长的数据量。 2. **分布式存储**：将单一文件划分为多个副本，分散在不同的节点上，实现了数据冗余和容错性，即使部分节点故障，也能保证数据的完整性。 3. **高吞吐量**：设计上考虑了大量并发客户端的访问，通过并行读写操作提高数据处理速度。 4. **数据一致性**：采用了类似BigTable的追加日志（log-structured merge trees）模型，确保了数据的一致性，即使在网络分区或故障恢复期间。 5. **简单接口**：为应用程序提供了一个统一、简单的API，降低了开发者在处理分布式文件系统复杂性的困扰。 6. **底层硬件兼容**：GFS 可以运行在标准的商用硬件上，降低了硬件成本和维护难度。 GFS 的成功实践不仅推动了 Google 内部的数据管理，也对整个IT业界产生了深远的影响，推动了分布式文件系统的发展，特别是在云计算和大数据时代。后续的研究和开源项目，如 Hadoop 分布式文件系统(HDFS)，都在一定程度上借鉴了 GFS 的设计理念和技术。

Legend:

Data messages

Control messages

Application

(file name, chunk index)

(chunk handle,

chunk locations)

GFS master

File namespace

/foo/bar

Instructions to chunkserver

Chunkserver state

GFS chunkserverGFS chunkserver

(chunk handle, byte range)

chunk data

chunk 2ef0

Linux file system Linux file system

GFS client

Figure 1: GFS Architecture

and replication decisions using global knowledge. However,

we must minimize its involvement in reads and writes so

that it does not become a bottleneck. Clients never read

and write ﬁle data through the master. Instead, a client asks

the master which chunkservers it should contact. It caches

this information for a limited time and interacts with the

chunkservers directly for many subsequent operations.

Let us explain the interactions for a simple read with refer-

ence to Figure 1. First, using the ﬁxed chunk size, the client

translates the ﬁle name and byte oﬀset speciﬁed by the ap-

plication into a chunk index within the ﬁle. Then, it sends

the master a request containing the ﬁle name and chunk

index. The master replies with the corresponding chunk

handle and locations of the replicas. The client caches this

information using the ﬁle name and chunk index as the key.

The client then sends a request to one of the replicas,

most likely the closest one. The request speciﬁes the chunk

handle and a byte range within that chunk. Further reads

of the same chunk require no more client-master interaction

until the cached information expires or the ﬁle is reopened.

In fact, the client typically asks for multiple chunks in the

same request and the master can also include the informa-

tion for chunks immediately following those requested. This

extra information sidesteps several future client-master in-

teractions at practically no extra cost.

2.5 Chunk Size

Chunk size is one of the key design parameters. We have

chosen 64 MB, which is much larger than typical ﬁle sys-

tem block sizes. Each chunk replica is stored as a plain

Linux ﬁle on a chunkserver and is extended only as needed.

Lazy space allocation avoids wasting space due to internal

fragmentation, perhaps the greatest objection against such

a large chunk size.

A large chunk size oﬀers several important advantages.

First, it reduces clients’ need to interact with the master

because reads and writes on the same chunk require only

one initial request to the master for chunk location informa-

tion. The reduction is especially signiﬁcant for our work-

loads because applications mostly read and write large ﬁles

sequentially. Even for small random reads, the client can

comfortably cache all the chunk location information for a

multi-TB working set. Second, since on a large chunk, a

client is more likely to perform many operations on a given

chunk, it can reduce network overhead by keeping a persis-

tent TCP connection to the chunkserver over an extended

period of time. Third, it reduces the size of the metadata

stored on the master. This allows us to keep the metadata

in memory, which in turn brings other advantages that we

will discuss in Section 2.6.1.

On the other hand, a large chunk size, even with lazy space

allocation, has its disadvantages. A small ﬁle consists of a

small number of chunks, perhaps just one. The chunkservers

storing those chunks may become hot spots if many clients

are accessing the same ﬁle. In practice, hot spots have not

been a major issue because our applications mostly read

large multi-chunk ﬁles sequentially.

However, hot spots did develop when GFS was ﬁrst used

by a batch-queue system: an executable was written to GFS

as a single-chunk ﬁle and then started on hundreds of ma-

chines at the same time. The few chunkservers storing this

executable were overloaded by hundreds of simultaneous re-

quests. We ﬁxed this problem by storing such executables

with a higher replication factor and by making the batch-

queue system stagger application start times. A potential

long-term solution is to allow clients to read data from other

clients in such situations.

2.6 Metadata

The master stores three major types of metadata: the ﬁle

and chunk namespaces, the mapping from ﬁles to chunks,

and the locations of each chunk’s replicas. All metadata is

kept in the master’s memory. The ﬁrst two types (names-

paces and ﬁle-to-chunk mapping) are also kept persistent by

logging mutations to an operation log stored on the mas-

ter’s local disk and replicated on remote machines. Using

a log allows us to update the master state simply, reliably,

and without risking inconsistencies in the event of a master

crash. The master does not store chunk location informa-

tion persistently. Instead, it asks each chunkserver about its

chunks at master startup and whenever a chunkserver joins

the cluster.

2.6.1 In-Memory Data Structures

Since metadata is stored in memory, master operations are

fast. Furthermore, it is easy and eﬃcient for the master to

periodically scan through its entire state in the background.

This periodic scanning is used to implement chunk garbage

collection, re-replication in the presence of chunkserver fail-

ures, and chunk migration to balance load and disk space

剩余14页未读，继续阅读

brianyu1983

粉丝: 0
资源: 1

Google File System：大规模分布式存储的创新设计与应用

Google File System Paper

Google三大篇著名论文Paper 机器学习AI必读 GFS MapReduce BigTable

Gfs Paper.rar

分布式系统学习——GFS谷歌文件系统Paper翻译1

Google-GFS,Bigtable,Mapreduce三大论文英文原版+中文翻译

google papers

Google云计算经典论文 GFS BigTable MapReduce

所有架构师都应该至少读上两遍的10篇论文-系统架构

数据库基础测验20241113.doc

微信小程序下拉选择组件

最新资源