![](https://csdnimg.cn/release/download_crawler_static/13772365/bg4.jpg)
protocols, deployment modes, or codebases. While the theory
of consensus has always allowed learners and acceptors to be
disaggregated, Delos is also the first production system that
can switch between converged and disaggregated acceptors.
In this paper, we make the following contributions:
•
We propose virtualizing consensus via the novel Vir-
tualLog and Loglet abstractions; and describe Delos, a
storage system that implements these abstractions.
•
Using production data, we show Delos upgrading to
NativeLoglet without downtime for a 10X latency im-
provement.
•
Using experiments, we show that Delos can: A) switch
to a disaggregated Loglet for a 10X improvement in
throughput under a 15ms p99 SLA; B) double its failure
threshold without lowering throughput via a Striped-
Loglet that rotates sequencers. Further, we show that
StripedLoglet can support over a million 1KB appends/s
on a log-only workload by sharding acceptors.
2 The Path to Virtual Consensus
Virtualization for faster deployment:
In 2017, Facebook
needed a table store for its core control plane services with
strong guarantees on durability, consistency, and availability.
Two practical imperatives drove the design and development
of this system: fast deployment (it had to reach production
within 6-9 months) and incremental evolution (it had to sup-
port better performance over time).
At the time, Facebook already operated four different stor-
age systems: a ZooKeeper service; a shared log service based
on LogDevice [3]; a key-value service called ZippyDB [5];
and a replicated MySQL service [13]. None of these systems
fit the exact use case, either due to a mismatch in API (e.g.,
ZooKeeper does not provide a table API) or fault-tolerance
guarantees (e.g., the MySQL service provided inadequate
availability).
Further, these systems could not be easily modified to pro-
vide the required API or guarantees. Each of them was a mono-
lith: the database API could not be easily changed, nor could
the underlying consensus protocol be re-used. In some sys-
tems, no abstraction boundary existed between the database
and the consensus protocol. In other systems, an abstraction
boundary did exist in the form of a shared log API, allowing
the underlying consensus protocol to be reused; however, no
support existed to migrate from one implementation of the
abstraction to another.
Building yet another monolithic system from scratch – in-
cluding a new consensus implementation – was not feasible
since we had to hit deployment within 6-9 months. Layering
the system over an existing shared log such as LogDevice
would allow us to reach production quickly, but also tie us for
perpetuity to the fault-tolerance and performance properties
of that consensus implementation.
Our solution was to virtualize consensus. In the remainder
of this paper, we describe how virtual consensus allowed us
to reach production quickly with an existing implementation
of consensus, and then migrate without downtime to new
implementations.
Virtualization for faster development:
Beyond fast ini-
tial deployment and online migration, virtualization also en-
abled faster development of new consensus implementations.
On its own, the shared log abstraction simplifies consensus-
based systems, separating applications from the logic of con-
sensus via a data-centric API. Virtualizing the shared log
further splits the consensus protocol into two layers: a control
plane, which includes the logic for reconfiguration, and a data
plane, which orders commands on the critical path.
In practice, such separation allowed us to incrementally
improve the system by re-implementing just the data plane of
the consensus protocol via new Loglets, while reusing the Vir-
tualLog control plane for features such as leader election and
membership changes. In the process, we completely changed
the operational characteristics and performance of the system,
as we describe later.
Importantly, such a separation also enables diversity in the
data plane. The last few years have seen a number of consen-
sus protocols with novel ordering mechanisms [3, 7,14
–
16,
22,24,31,35,37, 39,42], providing vastly different trade-offs
between performance and fault-tolerance. By making it easier
to implement such protocols and deploy them within running
systems, virtualization lowers the barrier to innovation.
3 Abstractions for Virtual Consensus
In this paper, we propose virtualizing consensus by virtual-
izing the shared log abstraction. We have three design goals
for virtualization. First, virtualization should be transparent
to applications , which should be unmodified and oblivious to
the virtualized nature of the log. Second, virtualization should
allow underlying logs to be simple and diverse, lowering
the barrier to new log implementations. Third, virtualization
should allow for migration between implementations without
downtime. We obtain these properties via two core abstrac-
tions.
In virtual consensus, the application operates above a Virtu-
alLog, which stitches together multiple independent Loglets.
The VirtualLog and Loglets expose a conventional shared log
API (see Figure 2). Applications can
append
an entry, receiv-
ing back a log position; call
checkTail
to obtain the first
unwritten position; call
readNext
to read the first entry in the
passed-in range; and call
prefixTrim
to indicate that a prefix
of the log can be trimmed. Virtualization requires two addi-
tions to this basic API: a
seal
command, which ensures that
any new appends fail; and an augmented
checkTail
response
USENIX Association 14th USENIX Symposium on Operating Systems Design and Implementation 619