f4: Facebook’s Warm BLOB Storage System
Subramanian Muralidhar
∗
, Wyatt Lloyd
†∗
, Sabyasachi Roy
∗
, Cory Hill
∗
, Ernest Lin
∗
, Weiwen Liu
∗
,
Satadru Pan
∗
, Shiva Shankar
∗
, Viswanath Sivakumar
∗
, Linpeng Tang
‡∗
, Sanjeev Kumar
∗
∗
Facebook Inc.,
†
University of Southern California,
‡
Princeton University
Abstract
Facebook’s corpus of photos, videos, and other Binary
Large OBjects (BLOBs) that need to be reliably stored
and quickly accessible is massive and continues to grow.
As the footprint of BLOBs increases, storing them in
our traditional storage system, Haystack, is becoming in-
creasingly inefficient. To increase our storage efficiency,
measured in the effective-replication-factor of BLOBs,
we examine the underlying access patterns of BLOBs
and identify temperature zones that include hot BLOBs
that are accessed frequently and warm BLOBs that are
accessed far less often. Our overall BLOB storage sys-
tem is designed to isolate warm BLOBs and enable us to
use a specialized warm BLOB storage system, f4. f4 is
a new system that lowers the effective-replication-factor
of warm BLOBs while remaining fault tolerant and able
to support the lower throughput demands.
f4 currently stores over 65PBs of logical BLOBs
and reduces their effective-replication-factor from 3.6
to either 2.8 or 2.1. f4 provides low latency; is resilient
to disk, host, rack, and datacenter failures; and provides
sufficient throughput for warm BLOBs.
1. Introduction
As Facebook has grown, and the amount of data shared
per user has grown, storing data efficiently has become
increasingly important. An important class of data that
Facebook stores is Binary Large OBjects (BLOBs),
which are immutable binary data. BLOBs are created
once, read many times, never modified, and sometimes
deleted. BLOB types at Facebook include photos, videos,
documents, traces, heap dumps, and source code. The
storage footprint of BLOBs is large. As of February 2014,
Facebook stored over 400 billion photos.
Haystack [
5
], Facebook’s original BLOB storage
system, has been in production for over seven years and is
designed for IO-bound workloads. It reduces the number
of disk seeks to read a BLOB to almost always one and
triple replicates data for fault tolerance and to support a
high request rate. However, as Facebook has grown and
evolved, the BLOB storage workload has changed. The
types of BLOBs stored have increased. The diversity in
size and create, read, and delete rates has increased. And,
most importantly, there is now a large and increasing
number of BLOBs with low request rates. For these
BLOBs, triple replication results in over provisioning
from a throughput perspective. Yet, triple replication also
provided important fault tolerance guarantees.
Our newer f4 BLOB storage system provides the
same fault tolerance guarantees as Haystack but at a
lower effective-replication-factor. f4 is simple, modular,
scalable, and fault tolerant; it handles the request rate
of BLOBs we store it in; it responds to requests with
sufficiently low latency; it is tolerant to disk, host, rack
and datacenter failures; and it provides all of this at a low
effective-replication-factor.
We describe f4 as a warm BLOB storage system
because the request rate for its content is lower than that
for content in Haystack and thus is not as “hot.” Warm
is also in contrast with cold storage systems [
20
,
40
]
that reliably store data but may take days or hours to
retrieve it, which is unacceptably long for user-facing
requests. We also describe BLOBs using temperature,
with hot BLOBs receiving many requests and warm
BLOBs receiving few.
There is a strong correlation between the age of
a BLOB and its temperature, as we will demonstrate.
Newly created BLOBs are requested at a far higher rate
than older BLOBs. For instance, the request rate for
week-old BLOBs is an order of magnitude lower than for
less-than-a-day old content for eight of nine examined
types. In addition, there is a strong correlation between
age and the deletion rate. We use these findings to inform
our design: the lower request rate of warm BLOBs en-
ables us to provision a lower maximum throughput for f4
than Haystack, and the low delete rate for warm BLOBs
enables us to simplify f4 by not needing to physically re-
claim space quickly after deletes. We also use our finding
to identify warm content using the correlation between
age and temperature.
Facebook’s overall BLOB storage architecture is de-
signed to enable warm storage. It includes a caching stack
that significantly reduces the load on the storage systems
and enables them to be provisioned for fewer requests
per BLOB; a transformer tier that handles computational-
intense BLOB transformation and can be scaled inde-
pendently of storage; a router tier that abstracts away
the underlying storage systems and enables seamless
migration between them; and the hot storage system,
Haystack, that aggregates newly created BLOBs into vol-
umes and stores them until their request and delete rates
have cooled off enough to be migrated to f4.
1
评论0