Windows Azure Storage：高可用云存储系统与一致性强的数据模型

分布式

需积分: 12 92 浏览量更新于2024-07-17 收藏 300KB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

Windows Azure Storage (WAS) 是一款高度可用的云存储服务，专为满足企业级数据存储需求而设计。该系统的核心理念是提供近乎无限的存储容量，让用户可以根据业务需求灵活地存储和管理数据，并按需付费。它支持三种主要的数据存储模型：Blob（文件）、Table（结构化存储）和Queue（消息传递），以适应不同类型的应用场景。 WAS架构的关键组成部分包括全球命名空间，这是统一管理所有存储资源的基础，允许用户通过一个单一的命名空间访问分布在不同地理位置的存储资源，提高了数据的一致性和可用性。数据模型方面，Blob存储适合非结构化的大量二进制数据，如图片、视频等；Table则适用于结构化的键值对数据，提供了强大的查询功能，适合关系数据库的替代方案；Queue则用于异步消息传递，确保数据在不同组件之间的可靠传递。资源的管理和扩展在WAS中至关重要。通过精细的资源预留和动态调整，系统能够根据用户的实际使用情况动态分配存储空间，确保性能和成本效益。此外，为了保证高可用性和容错性，WAS采用本地冗余和地理冗余两种复制策略，即使在发生故障时也能快速恢复数据，减少业务中断的风险。 WAS的负载均衡和复制系统也值得深入探讨。通过分布式计算和网络技术，WAS能实现数据的均衡分布，确保读写操作的高效处理。同时，数据在多个地理位置的复制有助于提高数据的可持久性和灾难恢复能力，降低了单点故障的影响。 Windows Azure Storage作为一个全面的云存储解决方案，不仅提供了强大的数据存储能力，还注重了数据的可靠性、一致性和扩展性。无论是对于个人开发者还是企业级应用，WAS都是一种灵活、经济且可靠的存储选择，极大地推动了云计算时代的数据管理和分析能力。

资源详情

资源推荐

3.3 Three Layers within a Storage Stamp

Also shown in Figure 1 are the three layers within a storage stamp. From bottom up these are:

Stream Layer – This layer stores the bits on disk and is in charge of distributing and replicating

the data across many servers to keep data durable within a storage stamp. The stream layer can be

thought of as a distributed file system layer within a stamp. It understands files, called “streams”

(which are ordered lists of large storage chunks called “extents”), how to store them, how to

replicate them, etc., but it does not understand higher level object constructs or their semantics. The

data is stored in the stream layer, but it is accessible from the partition layer. In fact, partition

servers (daemon processes in the partition layer) and stream servers are co-located on each storage

node in a stamp.

Partition Layer – The partition layer is built for (a) managing and understanding higher level data

abstractions (Blob, Table, Queue), (b) providing a scalable object namespace, (c) providing

transaction ordering and strong consistency for objects, (d) storing object data on top of the stream

layer, and (e) caching object data to reduce disk I/O.

Another responsibility of this layer is to achieve scalability by partitioning all of the data objects

within a stamp. As described earlier, all objects have a PartitionName; they are broken down into

disjointed ranges based on the PartitionName values and served by different partition servers. This

layer manages which partition server is serving what PartitionName ranges for Blobs, Tables, and

Queues. In addition, it provides automatic load balancing of PartitionNames across the partition

servers to meet the traffic needs of the objects.

Front-End (FE) layer – The Front-End (FE) layer consists of a set of stateless servers that take

incoming requests. Upon receiving a request, an FE looks up the AccountName, authenticates and

authorizes the request, then routes the request to a partition server in the partition layer (based on

the PartitionName). The system maintains a Partition Map that keeps track of the PartitionName

ranges and which partition server is serving which PartitionNames. The FE servers cache the

Partition Map and use it to determine which partition server to forward each request to. The FE

servers also stream large objects directly from the stream layer and cache frequently accessed data

for efficiency.

3.4 Two Replication Engines

Before describing the stream and partition layers in detail, we first give a brief overview of the two

replication engines in our system and their separate responsibilities.

Intra-Stamp Replication (stream layer) – This system provides synchronous replication and is

focused on making sure all the data written into a stamp is kept durable within that stamp. It keeps

enough replicas of the data across different nodes in different fault domains to keep data durable

within the stamp in the face of disk, node, and rack failures. Intra-stamp replication is done

completely by the stream layer and is on the critical path of the customer’s write requests. Once a

transaction has been replicated successfully with intra-stamp replication, success can be returned

back to the customer.

Inter-Stamp Replication (partition layer) – This system provides asynchronous replication and

is focused on replicating data across stamps. Inter-stamp replication is done in the background and

is off the critical path of the customer’s request. This replication is at the object level, where either

the whole object is replicated or recent delta changes are replicated for a given account. Inter-

stamp replication is used for (a) keeping a copy of an account’s data in two locations for disaster

recovery and (b) migrating an account’s data between stamps. Inter-stamp replication is configured

for an account by the location service and performed by the partition layer.

Inter-stamp replication is focused on replicating objects and the transactions applied to those

objects, whereas intra-stamp replication is focused on replicating blocks of disk storage that are

used to make up the objects.

We separated replication into intra-stamp and inter-stamp at these two different layers for the

following reasons. Intra-stamp replication provides durability against hardware failures, which

occur frequently in large scale systems, whereas inter-stamp replication provides geo-redundancy

against geo-disasters, which are rare. It is crucial to provide intra-stamp replication with low

latency, since that is on the critical path of user requests; whereas the focus of inter-stamp

replication is optimal use of network bandwidth between stamps while achieving an acceptable

level of replication delay. They are different problems addressed by the two replication schemes.

Another reason for creating these two separate replication layers is the namespace each of these

two layers has to maintain. Performing intra-stamp replication at the stream layer allows the

amount of information that needs to be maintained to be scoped by the size of a single storage

stamp. This focus allows all of the meta-state for intra-stamp replication to be cached in memory

for performance (see Section

4), enabling WAS to provide fast replication with strong consistency

by quickly committing transactions within a single stamp for customer requests. In contrast, the

partition layer combined with the location service controls and understands the global object

namespace across stamps, allowing it to efficiently replicate and maintain object state across data

centers.

4. Stream Layer

The stream layer provides an internal interface used only by the partition layer. It provides a file

system like namespace and API, except that all writes are append-only. It allows clients (the

partition layer) to open, close, delete, rename, read, append to, and concatenate these large files,

which are called streams. A stream is an ordered list of extent pointers, and an extent is a sequence

of append blocks.

Figure 2 shows stream “//foo”, which contains (pointers to) four extents (E1, E2, E3, and E4).

Each extent contains a set of blocks that were appended to it. E1, E2 and E3 are sealed extents. It

means that they can no longer be appended to; only the last extent in a stream (E4) can be appended

to. If an application reads the data of the stream from beginning to end, it would get the block

contents of the extents in the order of E1, E2, E3 and E4.

Stream //foo

Pointer to Extent E1

….. B

Pointer to Extent E2

….. B

Extent E1 - Sealed

Extent E2 - Sealed

Pointer to Extent E3

….. B

Extent E3 - Sealed

Pointer to Extent E4

Extent E4 - Unsealed

Figure 2: Example stream with four extents

In more detail these data concepts are:

Block – This is the minimum unit of data for writing and reading. A block can be up to N bytes

(e.g. 4MB). Data is written (appended) as one or more concatenated blocks to an extent, where

blocks do not have to be the same size. The client does an append in terms of blocks and controls

the size of each block. A client read gives an offset to a stream or extent, and the stream layer

reads as many blocks as needed at the offset to fulfill the length of the read. When performing a

read, the entire contents of a block are read. This is because the stream layer stores its checksum

validation at the block level, one checksum per block. The whole block is read to perform the

checksum validation, and it is checked on every block read. In addition, all blocks in the system

are validated against their checksums once every few days to check for data integrity issues.

Extent – Extents are the unit of replication in the stream layer, and the default replication policy is

to keep three replicas within a storage stamp for an extent. Each extent is stored in an NTFS file

and consists of a sequence of blocks. The target extent size used by the partition layer is 1GB. To

store small objects, the partition layer appends many of them to the same extent and even in the

same block; to store large TB-sized objects (Blobs), the object is broken up over many extents by

the partition layer. The partition layer keeps track of what streams, extents, and byte offsets in the

extents in which objects are stored as part of its index.

剩余25页未读，继续阅读

tianyuanzk

粉丝: 0
资源: 7

Windows Azure Storage：高可用云存储系统与一致性强的数据模型

AzureStorage

Windows Azure Storage（讲稿）

Erasure Coding in Windows Azure Storage.pdf

Windows Azure Storage：高可用云存储服务

Windows Azure Storage中Erasure Coding原理与应用

Python云存储：S3、Azure Blob Storage的性能与成本分析

WindowsAzure.StorageClient API

设计一段Python，要求将azure storage account的指定容器中的指定文件夹里的Excel文件转换为CSV文件

1.Azure Storage 账户如何在c#中配置 2.如何在文件中读取配置文件内的连接字符串

查找一个Azure的本地文件upload上传示例

java代码查找一个Azure的本地文件upload上传示例

Java连接Azure blob，设置超时时间

使用springboot搭建一个上传视频的功能

blob视频下载python

windows2019 文件夹同步

类型“BlockBlobClient”上不存在属性“uploadData”。ts

windows vscode 安装GDAL

sql server怎么读取location指向gen2 parquet文件的外部表

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask. Hive warehouse is non-local, but /home/hw13989666127/HiveFunction-jar-with-dependencies.jar specifies file on local filesystem. Resources on non-local warehouse should specify a non-local scheme/path

最新资源