These emulators provide increasing numbers of pa-
rameters and mechanisms to recreate different network
behaviors, such as traffic shapers, policers, queue disci-
plines, stochastic i.i.d. or autocorrelated loss, reordering,
bit errors, and MAC dynamics. However, properly setting
these parameters to emulate a particular target network
remains an open problem.
One line of work has focused on improving emulator
precision in terms of the level of detail and fidelity at
modeling small-scale effects (e.g., “Two aspects influence
the accuracy of an emulator: how detailed is the model of
the system, and how closely the hardware and software
can reproduce the timing computed by the model” [
6
]).
Pantheon takes a different approach, instead focusing on
accuracy in terms of how well an emulator recreates the
performance of a set of transport algorithms.
Congestion control.
Internet congestion control has a
deep literature. The original DECBit [
32
] and Tahoe [
22
]
algorithms responded to one-bit feedback from the net-
work, increasing and decreasing a congestion window in
response to acknowledgments and losses. More recently,
researchers have tried to formalize the protocol-design
process by generating a congestion-control scheme as a
function of an objective function and prior beliefs about
the network and workload. Remy [
37
,
41
] and PCC [
12
]
are different kinds of “learned” schemes [
35
]. Remy uses
an offline optimizer that generates a decision tree to opti-
mize a global utility score based on network simulations.
PCC uses an online optimizer that adapts its sending rate
to maximize a local utility score in response to packet
losses and RTT samples. In our current work (§ 6), we
ask whether it is possible to quickly train an algorithm
from first principles to produce good global performance
on real Internet paths.
3 Design and implementation
This section describes the design and implementation of
the Pantheon, a system that automatically measures the
performance of many transport protocols and congestion-
control schemes across a diverse set of network paths. By
allowing the community to repeatably evaluate transport
algorithms in scripted comparative tests across real-world
network paths, posted to a public archive of results, the
Pantheon aims to help researchers develop and test algo-
rithms more rapidly and reproducibly.
Below, we demonstrate several uses for Pantheon: com-
paring existing congestion-control schemes on real-world
networks (§4); calibrating network emulators that accu-
rately reproduce real-world performance (§5); and design-
ing and testing new congestion-control schemes (§6).
Label Scheme LoC
Copa Copa [2] 46
LEDBAT LEDBAT/µTP [36] (libutp) 48
PCC PCC
†
[12] 46
QUIC QUIC Cubic [24] (proto-quic) 119
SCReAM SCReAM [23] 541
Sprout Sprout
†
[42] 46
Tao RemyCC “100x” (2014) [37] 43
BBR TCP BBR [7] 52
Cubic TCP Cubic [18] (Linux default) 30
Vegas TCP Vegas [5] 50
Verus Verus
†
[43] 43
— Vivace [13] 37
WebRTC WebRTC media [4] in Chromium 283
— FillP (work in progress) 41
Indigo LSTM neural network (work in progress) 35
Figure 1: The Pantheon’s transport schemes (§3.1.1) and
the labels used for them in figures in this paper. Shown
are the number of lines of Python, C++, or Javascript code
in each wrapper that implements the common abstraction.
Schemes marked † are modified to reduce MTU.
3.1 Design overview
Pantheon has three components: (1) a software repository
containing pointers to transport-protocol implementations,
each wrapped to expose a common testing interface based
on the abstraction of a full-throttle flow; (2) testing infras-
tructure that runs transport protocols in scripted scenarios,
instruments the network to log when each packet was sent
and received, and allows flows to be initiated by nodes be-
hind a network address translator (NAT); and (3) a global
observatory of network nodes, enabling measurements
across a wide variety of paths. We describe each in turn.
3.1.1 A collection of transport algorithms, each ex-
posing the same interface
To test each transport protocol or congestion-control
scheme on equal footing, Pantheon requires it to expose a
common abstraction for testing: a full-throttle flow that
runs until a sender process is killed. The simplicity of this
interface has allowed us (and a few external contributors
so far) to write simple wrappers for a variety of schemes
and contribute them to the Pantheon, but limits the kinds
of evaluations the system can do.
1
Figure 1 lists the currently supported schemes, plus
the size (in lines of code) of a wrapper script to expose
the required abstraction. For all but three schemes, no
modification was required to the existing implementation.
The remaining three had a hard-coded MTU size and
1
For example, the interface allows measurements of combinations
of long-running flows (with timed events to start and stop a flow), but
does not allow the caller to run a scheme until it has transferred exactly
x
bytes. This means that the Pantheon cannot reliably measure the flow-
completion time of a mix of small file transfers.