【免费】f4:Facebook’sWarmBLOBStorageSystem

facebook

BLOB

需积分: 0 26 浏览量更新于2023-03-16 评论收藏 665KB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

资源详情

资源评论

资源推荐

f4: Facebook’s Warm BLOB Storage System

Subramanian Muralidhar

∗

, Wyatt Lloyd

†∗

, Sabyasachi Roy

∗

, Cory Hill

∗

, Ernest Lin

∗

, Weiwen Liu

∗

Satadru Pan

∗

, Shiva Shankar

∗

, Viswanath Sivakumar

∗

, Linpeng Tang

‡∗

, Sanjeev Kumar

∗

Facebook Inc.,

†

University of Southern California,

‡

Princeton University

Abstract

Facebook’s corpus of photos, videos, and other Binary

Large OBjects (BLOBs) that need to be reliably stored

and quickly accessible is massive and continues to grow.

As the footprint of BLOBs increases, storing them in

our traditional storage system, Haystack, is becoming in-

creasingly inefﬁcient. To increase our storage efﬁciency,

measured in the effective-replication-factor of BLOBs,

we examine the underlying access patterns of BLOBs

and identify temperature zones that include hot BLOBs

that are accessed frequently and warm BLOBs that are

accessed far less often. Our overall BLOB storage sys-

tem is designed to isolate warm BLOBs and enable us to

use a specialized warm BLOB storage system, f4. f4 is

a new system that lowers the effective-replication-factor

of warm BLOBs while remaining fault tolerant and able

to support the lower throughput demands.

f4 currently stores over 65PBs of logical BLOBs

and reduces their effective-replication-factor from 3.6

to either 2.8 or 2.1. f4 provides low latency; is resilient

to disk, host, rack, and datacenter failures; and provides

sufﬁcient throughput for warm BLOBs.

1. Introduction

As Facebook has grown, and the amount of data shared

per user has grown, storing data efﬁciently has become

increasingly important. An important class of data that

Facebook stores is Binary Large OBjects (BLOBs),

which are immutable binary data. BLOBs are created

once, read many times, never modiﬁed, and sometimes

deleted. BLOB types at Facebook include photos, videos,

documents, traces, heap dumps, and source code. The

storage footprint of BLOBs is large. As of February 2014,

Facebook stored over 400 billion photos.

Haystack [

], Facebook’s original BLOB storage

system, has been in production for over seven years and is

designed for IO-bound workloads. It reduces the number

of disk seeks to read a BLOB to almost always one and

triple replicates data for fault tolerance and to support a

high request rate. However, as Facebook has grown and

evolved, the BLOB storage workload has changed. The

types of BLOBs stored have increased. The diversity in

size and create, read, and delete rates has increased. And,

most importantly, there is now a large and increasing

number of BLOBs with low request rates. For these

BLOBs, triple replication results in over provisioning

from a throughput perspective. Yet, triple replication also

provided important fault tolerance guarantees.

Our newer f4 BLOB storage system provides the

same fault tolerance guarantees as Haystack but at a

lower effective-replication-factor. f4 is simple, modular,

scalable, and fault tolerant; it handles the request rate

of BLOBs we store it in; it responds to requests with

sufﬁciently low latency; it is tolerant to disk, host, rack

and datacenter failures; and it provides all of this at a low

effective-replication-factor.

We describe f4 as a warm BLOB storage system

because the request rate for its content is lower than that

for content in Haystack and thus is not as “hot.” Warm

is also in contrast with cold storage systems [

]

that reliably store data but may take days or hours to

retrieve it, which is unacceptably long for user-facing

requests. We also describe BLOBs using temperature,

with hot BLOBs receiving many requests and warm

BLOBs receiving few.

There is a strong correlation between the age of

a BLOB and its temperature, as we will demonstrate.

Newly created BLOBs are requested at a far higher rate

than older BLOBs. For instance, the request rate for

week-old BLOBs is an order of magnitude lower than for

less-than-a-day old content for eight of nine examined

types. In addition, there is a strong correlation between

age and the deletion rate. We use these ﬁndings to inform

our design: the lower request rate of warm BLOBs en-

ables us to provision a lower maximum throughput for f4

than Haystack, and the low delete rate for warm BLOBs

enables us to simplify f4 by not needing to physically re-

claim space quickly after deletes. We also use our ﬁnding

to identify warm content using the correlation between

age and temperature.

Facebook’s overall BLOB storage architecture is de-

signed to enable warm storage. It includes a caching stack

that signiﬁcantly reduces the load on the storage systems

and enables them to be provisioned for fewer requests

per BLOB; a transformer tier that handles computational-

intense BLOB transformation and can be scaled inde-

pendently of storage; a router tier that abstracts away

the underlying storage systems and enables seamless

migration between them; and the hot storage system,

Haystack, that aggregates newly created BLOBs into vol-

umes and stores them until their request and delete rates

have cooled off enough to be migrated to f4.

Web Tier!

User Requests (Browsers, Mobile Devices)!

CDN!

Graph!

Store!

BLOB Storage System!

C1 R1

C3 R2

Figure 1: Reading (

R1-R4

) and creating (

C1-C3

) BLOBs.

f4 stores volumes of warm BLOBs in cells that use dis-

tributed erasure coding, which uses fewer physical bytes

than triple replication. It uses Reed-Solomon(10,4) [

]

coding and lays blocks out on different racks to ensure

resilience to disk, machine, and rack failures within a

single datacenter. Is uses XOR coding in the wide-area

to ensure resilience to datacenter failures. f4 has been

running in production at Facebook for over 19 months.

f4 currently stores over 65PB of logical data and saves

over 53PB of storage.

Our contributions in this paper include:

•

A case for warm storage that informs future research

on it and justiﬁes our efforts.

•

The design of our overall BLOB storage architecture

that enables warm storage.

•

The design of f4, a simple, efﬁcient, and fault tolerant

warm storage solution that reduces our effective-

replication-factor from 3.6 to 2.8 and then to 2.1.

•

A production evaluation of f4.

The paper continues with background in Section 2.

Section 3 presents the case for warm storage. Section 4

presents the design of our overall BLOB storage archi-

tecture that enables warm storage. f4 is described in

Section 5. Section 6 covers a production evaluation of

f4, Section 7 covers lessons learned, Section 8 covers

related work, and Section 9 concludes.

2. Background

This section explains where BLOB storage ﬁts in the full

architecture of Facebook. It also describes the different

types of BLOBs we store and their size distributions.

2.1 Where BLOB Storage Fits

Figure 1 shows how BLOB storage ﬁts into the overall

architecture at Facebook. BLOB creates—e.g., a video

upload—originate on the web tier (

). The web tier

writes the data to the BLOB storage system (

) and

then stores the handle for that data into our graph store

(

), Tao [

]. The handle can be used to retrieve or delete























































Figure 2: Size distribution for ﬁve BLOB types.

the BLOB. Tao associates the handle with other elements

of the graph, e.g., the owner of a video.

BLOB reads—e.g., watching a video—also originate

on the web tier (

). The web tier accesses the Graph

Store (

) to ﬁnd the necessary handles and constructs

a URL that can be used to fetch the BLOB. When

the browser later sends a request for the BLOB (

the request ﬁrst goes to a content distribution network

(CDN) [

] that caches commonly accessed BLOBs.

If the CDN does not have the requested BLOB, it sends

a request to the BLOB storage system (

), caches the

BLOB, and returns it to the user. The CDN shields the

storage system from a signiﬁcant number of requests on

frequently accessed data, and we return to its importance

in Sections 4.1.

2.2 BLOBs Explained

BLOBs are immutable binary data. They are created once,

read potentially many times, and can only be deleted, not

modiﬁed. This covers many types of content at Facebook.

Most BLOB types are user facing, such as photos, videos,

and documents. Other BLOB types are internal, such as

traces, heap dumps, and source code. User-facing BLOBs

are more prevalent so we focus on them for the remainder

of the paper and refer to them as simply BLOBs.

Figure 2 shows the distribution of sizes for ﬁve types

of BLOBs. There is a signiﬁcant amount of diversity

in the sizes of BLOBs, which has implications for our

design as discussed in Section 5.6.

3. The Case for Warm Storage

This section motivates the creation of a warm storage

system at Facebook. It demonstrates that temperature

zones exist, age is a good proxy for temperature, and that

warm content is large and growing.

Methodology

The data presented in this section is

derived from a two-week trace, benchmarks of existing

systems, and daily snapshots of summary statistics. The

trace includes a random 0.1% of reads, 10% of creates,

and 10% of deletes.

   





         











Figure 3: Relative request rates by age. Each line is rela-

tive to only itself, absolute values have been denormal-

ized to increase readability, and points mark an order-of-

magnitude decrease in request rate.

Data is presented for nine user-facing BLOB types.

We exclude some data types from some analysis due to

incomplete logging information.

The nine BLOB types include Proﬁle Photos, Photos,

HD Photos, Mobile Sync Photos [

], HD Mobile Sync

Photos, Group Attachments [

], Videos, HD Videos,

and Message (chat) Attachments. Group Attachments

and Message Attachments are opaque BLOBS to our

storage system, they can be text, pdfs, presentation, etc.

Temperature Zones Exist

To make the case for warm

storage we ﬁrst show that temperature zones exist, i.e.,

that content begins as hot, receiving many requests, and

then cools over time, receiving fewer and fewer requests.

Figure 3 shows the relative request rate, requests-per-

object-per-hour, for content of a given age. The two-week

trace of 0.1% of reads was used to create this ﬁgure. The

age of each object being read is recorded and these are

bucketed into 1-day intervals. We then count the number

of requests to the daily buckets for each hour in the trace

and report the mean—the medians are similar but noisier.

Absolute values are denormalized to increase readability

so each line is relative to only itself. Points mark order-

of-magnitude decreases.

The existence of temperature zones is clear in the

trend of decreasing request rates over time. For all nine

types, content less than one day old receives more than

100 times the request rate of one-year-old content. For

eight of the types the request rate drops by an order of

magnitude in less than a week, and for six of the types

the request rate drops by 100x in less than 60 days.

Differentiating Temperature Zones

Given that tem-

perature zones exist, the next questions to answer are

how to differentiate warm from hot content and when it

is safe to move content to warm storage. We deﬁne the









   









         











Figure 4: 99th percentile load in IOPS/TB of data for

different BLOB types for BLOBs of various ages.

warm temperature zone to include unchanging content

with a low request rate. BLOBs are not modiﬁed, so the

only changes are the deletes. Thus, differentiating warm

from hot content depends on the request and delete rates.

First, we examine the request rate. To determine

where to draw the line between hot and warm storage

we consider near-worst-case request rates because our

internal service level objects require low near-worst-case

latency during our busiest periods of the day.

Figure 4 shows the 99th percentile or near-worst-case

request load for BLOBs of various types grouped by age.

The two-week trace of 0.1% of reads was used to create

this ﬁgure. The age of each object read is recorded and

these are bucketed into intervals equivalent to the time

needed to create 1 TB of that BLOB type. For instance, if

1 TB of a type is created every 3600 seconds, then the ﬁrst

bucket is for ages of 0-3599 seconds, the second is for

3600-7200 seconds, and so on.

We then compensate for

the 0.1% sampling rate by looking at windows of 1000

seconds. We report the 99th percentile request rate for

these windows, i.e., we report the 99th percentile count

of requests in a 1000 second window across our two-

week trace for each age bucket. The 4TB disks used in f4

can deliver a maximum of 80 Input/Output Operations

Per Second (IOPS) while keeping per-request latency

acceptably low. The ﬁgure shows this peak warm storage

throughput at 20 IOPS/TB.

For seven of the nine types the near-worst-case

throughput is below the capacity of the warm storage sys-

tem in less than a week. For Photos, it takes

3 months to

drop below the capacity of warm storage and for Proﬁle

Photos it takes a year.

We also examined, but did not rigorously quantify, the

deletion rate of BLOB types over time. The general trend

We spread newly created BLOBs over many hosts and disks, so no

host or disk in our system is subject to the extreme loads on far left of

Figure 4. We elaborate on this point further in Section 5.6











  

















































Figure 5: Median percentage of each type that was warm

9-6 months ago, 6-3 months ago, 3 months ago to now.

The remaining percentage of each type is hot.

is that most deletes are for young BLOBs and that once

the request rate for a BLOB drops below the threshold

of warm storage, the delete rate is low as well.

Combining the deletion analysis with the request rate

analysis yields an age of a month as a safe delimiter

between hot and warm content for all but two BLOB

types. One type, Proﬁle Photos, is not moved to warm

storage. The other, Photos, uses a three months threshold.

Warm Content is Large and Growing

We ﬁnish the

case for warm storage by demonstrating that the percent-

age of content that is warm is large and continuing to

grow. Figure 5 gives the percentage of content that is

warm for three-month intervals for six BLOB types.

We use the above analysis to determine the warm

cutoff for each type, i.e., one month for most types.

This ﬁgure reports the median percentage of content for

each type that is warm in three-month intervals from 9-6

months ago, 6-3 months ago, and 3 months ago to now.

The ﬁgure shows that warm content is a large percent-

age of all objects: in the oldest interval more than 80%

of objects are warm for all types. It also shows that the

warm fraction is increasing: in the most recent interval

more than 89% of objects are warm for all types.

This section showed that temperature zones exist, that

the line between hot and warm content can safely be

drawn for existing types at Facebook at one month

for most types, and that warm content is a large and

growing percentage of overall BLOB content. Next,

we describe how Facebook’s overall BLOB storage

architecture enables warm storage.

4. BLOB Storage Design

Our BLOB storage design is guided by the principle of

keeping components simple, focused, and well-matched

to their job. In this section we explain volumes, describe

Transformer

Tier!

Router Tier!

Haystack!

Hot Storage!

f4!

Warm Storage!

Controller!

CDN!

Figure 6: Overall BLOB Storage Architecture with cre-

ates (

C1-C2

), deletes (

D1-D2

), and reads (

R1-R4

). Cre-

ates are handled by Haystack, most deletes are handled

by Haystack, reads are handled by either Haystack or f4.

the full design of our BLOB storage system, and explain

how it enables focused and simple warm storage with f4.

Volumes

We aggregate BLOBs together into logical

volumes. Volumes aggregate ﬁlesystem metadata, allow-

ing our storage systems to waste few IOPS as we discuss

further below. We categorize logical volumes into two

classes. Volumes are initially unlocked and support reads,

creates (appends), and deletes. Once volumes are full,

at around 100GB in size, they transition to being locked

and no longer allow creates. Locked volumes only allow

reads and deletes.

Each volume is comprised of three ﬁles: a data ﬁle, an

index ﬁle, and a journal ﬁle. The data ﬁle and index ﬁles

are the same as the published version of Haystack [

while the journal ﬁle is new. The data ﬁle holds each

BLOB along with associated metadata such as the key,

the size, and checksum. The index ﬁle is a snapshot of the

in-memory lookup structure of the storage machines. Its

main purpose is allowing rebooted machines to quickly

reconstruct their in-memory indexes. The journal ﬁle

tracks BLOBs that have been deleted; whereas in the

original version of Haystack, deletes were handled by

updating the data and index ﬁles directly. For locked

volumes, the data and index ﬁles are read-only, while the

journal ﬁle is read-write. For unlocked volumes, all three

ﬁles are read-write.

4.1 Overall Storage System

The full BLOB storage architecture is shown in Figure 6.

Creates enter the system at the router tier (

) and

are directed to the appropriate host in the hot storage

system (

). Deletes enter the system at the router

tier (

) and are directed to the appropriate hosts in

appropriate storage system (

). Reads enter the system

at the caching stack (

) and, if not satisﬁed there,

traverse through the transformer tier (

) to the router

tier (

) that directs them to the appropriate host in the

appropriate storage system (R4).

剩余15页未读，继续阅读

miller_lover

粉丝: 37
资源: 10

会员权益专享

f4: Facebook’s Warm BLOB Storage System

评论0

会员权益专享

最新资源

f4: Facebook’s Warm BLOB Storage System

评论0

azure-storage-blob-11.0.0.jar

ORACLE数据库中主要字段类型的读写例子(包括：Long、Raw、Blob).docx

ORACLE数据库中主要字段类型的读写例子(包括：Long、Raw、Blob).pdf

Magick::Blob m_blob;用QImage表示

torch::from_blob用法

pcl::fromPCLPointCloud2 (blob, cloud);

torch::from_blob（）

java流式读取Oracle,Java：从Oracle读取Blob

Azure Blob Storage

pcl::fromPCLPointCloud2 (blob, cloud);仅用于pcd_v7吗

python的blobfile库怎么用

创建账号Azure Blob Storage

使用 Microsoft:registered: Azure:registered: 存储 Blob 的 3 种方法：使用 HTTPS 和 WASB-matlab开发

ORACLE数据库中主要字段类型的读写例子(包括：Long、Raw、Blob) (2).docx

ORACLE数据库中主要字段类型的读写例子(包括：Long、Raw、Blob) (2).pdf

会员权益专享

最新资源