Google File System (GFS)：大规模分布式存储与高并发设计

需积分: 10 151 浏览量更新于2024-07-27 收藏 269KB PDF 举报

《Google File System (GFS) - SOSP 2003》是一篇介绍Google公司内部研发的分布式文件系统的关键论文。该系统是为大规模、数据密集型的分布式应用设计的，旨在提供高可用性和性能，即使在廉价的商业硬件上也能运行。与早期的分布式文件系统设计理念有所不同，GFS的设计受到了Google特定的应用工作负载和技术环境的影响。 GFS的重要贡献在于其对传统文件系统假设的挑战和创新设计。它解决了存储容量需求，广泛应用于Google内部，支持服务的数据生成和处理，以及研究和开发工作，这些工作需要海量的数据集。截至报告撰写时，最大的集群已部署在数千台机器上，跨越数千个磁盘，存储容量达到数百太字节，显示了其强大的扩展性和性能。以下是GFS的一些关键特性： 1. **分布式存储**：GFS采用分布式架构，将大量数据分散存储在多个节点上，而非集中式管理，这样可以提高系统的容错能力和数据的并行访问能力。这种设计有助于避免单点故障，并能支持大规模并发读写操作。 2. **高可用性**：为了应对硬件故障，GFS采用了复制策略，确保数据在多个节点上有备份。通过冗余存储和心跳检测机制，系统能够在出现故障时自动恢复服务，保证了系统的持续运行。 3. **性能优化**：针对Google应用的工作负载特性，GFS优化了文件分配和访问策略，比如大块连续分配（chunking）和预读取，以减少I/O开销和延迟。同时，它也采用了异步复制技术，允许数据写入时立即返回响应，提高了写入吞吐量。 4. **成本效益**：利用 commodity hardware，GFS降低了存储和运维的成本，适合大规模部署在经济高效的硬件上。这使得大型数据处理变得经济可行，而无需依赖昂贵的专业级存储解决方案。 5. **适应性**：GFS的设计充分考虑了Google当时的环境和技术趋势，比如快速增长的数据量、动态变化的工作负载需求以及未来的技术发展。这种灵活性使它能够随着业务的扩展和需求的变化进行适应。 6. **规模和扩展性**：GFS能够轻松扩展到数千台机器和数千个磁盘，这意味着它能够支撑Google内部各种大规模的项目，从搜索索引到机器学习训练等。总结来说，Google File System (GFS) 是一个创新的分布式文件系统，它在设计上充分考虑了实际应用需求和技术环境，从而实现了高可用性、高性能和低成本。通过在大规模数据处理场景下的成功应用，GFS为后来的分布式存储系统设计提供了重要的参考和启示。

Legend:

Data messages

Control messages

Application

(file name, chunk index)

(chunk handle,

chunk locations)

GFS master

File namespace

/foo/bar

Instructions to chunkserver

Chunkserver state

GFS chunkserverGFS chunkserver

(chunk handle, byte range)

chunk data

chunk 2ef0

Linux file system Linux file system

GFS client

Figure 1: GFS Architecture

and replication decisions using global knowledge. However,

we must minimize its involvement in reads and writes so

that it does not become a bottleneck. Clients never read

and write ﬁle data through the master. Instead, a client asks

the master which chunkservers it should contact. It caches

this information for a limited time and interacts with the

chunkservers directly for many subsequent operations.

Let us explain the interactions for a simple read with refer-

ence to Figure 1. First, using the ﬁxed chunk size, the client

translates the ﬁle name and byte oﬀset speciﬁed by the ap-

plication into a chunk index within the ﬁle. Then, it sends

the master a request containing the ﬁle name and chunk

index. The master replies with the corresponding chunk

handle and locations of the replicas. The client caches this

information using the ﬁle name and chunk index as the key.

The client then sends a request to one of the replicas,

most likely the closest one. The request speciﬁes the chunk

handle and a byte range within that chunk. Further reads

of the same chunk require no more client-master interaction

until the cached information expires or the ﬁle is reopened.

In fact, the client typically asks for multiple chunks in the

same request and the master can also include the informa-

tion for chunks immediately following those requested. This

extra information sidesteps several future client-master in-

teractions at practically no extra cost.

2.5 Chunk Size

Chunk size is one of the key design parameters. We have

chosen 64 MB, which is much larger than typical ﬁle sys-

tem block sizes. Each chunk replica is stored as a plain

Linux ﬁle on a chunkserver and is extended only as needed.

Lazy space allocation avoids wasting space due to internal

fragmentation, perhaps the greatest objection against such

a large chunk size.

A large chunk size oﬀers several important advantages.

First, it reduces clients’ need to interact with the master

because reads and writes on the same chunk require only

one initial request to the master for chunk location informa-

tion. The reduction is especially signiﬁcant for our work-

loads because applications mostly read and write large ﬁles

sequentially. Even for small random reads, the client can

comfortably cache all the chunk location information for a

multi-TB working set. Second, since on a large chunk, a

client is more likely to perform many operations on a given

chunk, it can reduce network overhead by keeping a persis-

tent TCP connection to the chunkserver over an extended

period of time. Third, it reduces the size of the metadata

stored on the master. This allows us to keep the metadata

in memory, which in turn brings other advantages that we

will discuss in Section 2.6.1.

On the other hand, a large chunk size, even with lazy space

allocation, has its disadvantages. A small ﬁle consists of a

small number of chunks, perhaps just one. The chunkservers

storing those chunks may become hot spots if many clients

are accessing the same ﬁle. In practice, hot spots have not

been a major issue because our applications mostly read

large multi-chunk ﬁles sequentially.

However, hot spots did develop when GFS was ﬁrst used

by a batch-queue system: an executable was written to GFS

as a single-chunk ﬁle and then started on hundreds of ma-

chines at the same time. The few chunkservers storing this

executable were overloaded by hundreds of simultaneous re-

quests. We ﬁxed this problem by storing such executables

with a higher replication factor and by making the batch-

queue system stagger application start times. A potential

long-term solution is to allow clients to read data from other

clients in such situations.

2.6 Metadata

The master stores three major types of metadata: the ﬁle

and chunk namespaces, the mapping from ﬁles to chunks,

and the locations of each chunk’s replicas. All metadata is

kept in the master’s memory. The ﬁrst two types (names-

paces and ﬁle-to-chunk mapping) are also kept persistent by

logging mutations to an operation log stored on the mas-

ter’s local disk and replicated on remote machines. Using

a log allows us to update the master state simply, reliably,

and without risking inconsistencies in the event of a master

crash. The master does not store chunk location informa-

tion persistently. Instead, it asks each chunkserver about its

chunks at master startup and whenever a chunkserver joins

the cluster.

2.6.1 In-Memory Data Structures

Since metadata is stored in memory, master operations are

fast. Furthermore, it is easy and eﬃcient for the master to

periodically scan through its entire state in the background.

This periodic scanning is used to implement chunk garbage

collection, re-replication in the presence of chunkserver fail-

ures, and chunk migration to balance load and disk space

剩余14页未读，继续阅读

honXian

粉丝: 0
资源: 7

Google File System (GFS)：大规模分布式存储与高并发设计

Google三大论文英文原版

Google三篇论文-GFS英文版.pdf

13、GFS.pdf论文

Google Mapreduce,GFS,Bigtable--Google三大核心技术论文

gfs+BigTable+map reduce 三合一.zip

Google三大篇著名论文Paper 机器学习AI必读 GFS MapReduce BigTable

Hadoop - 权威网站和经典书籍

【机器人】将ChatGPT飞书机器人钉钉机器人企业微信机器人公众号部署到vercel及docker_pgj.zip

图数据分析中基于对比学习的异常检测算法的Python实现及应用-含代码及详细解释说明

专题调研登记表.docx

最新资源