Initial End-to-End Performance Evaluation of 10-Gigabit Ethernet
Justin (Gus) Hurwitz, Wu-chun Feng
ghurwitz,feng
@lanl.gov
Research & Development in Advanced Network Technology (RADIANT)
Computer & Computational Sciences Division
Los Alamos National Laboratory
Los Alamos, NM 87545
Abstract
We present an initial end-to-end performance evaluation
of Intel’s
R
10-Gigabit Ethernet (10GbE) network inter-
face card (or adapter). With appropriate optimizations to
the configurations of Linux, TCP, and the 10GbE adapter,
we achieve over 4-Gb/s throughput and 21-
s end-to-end
latency between applications in a local-area network de-
spite using less capable, lower-end PCs. These results in-
dicate that 10GbE may also be a cost-effective solution for
system-area networks in commodity clusters, data centers,
and web-server farms as well as wide-area networks in sup-
port of computational and data grids.
1. Introduction
From its humble beginnings as shared Ethernet to its
current success as switched Ethernet in local- and system-
area networks (LANs and SANs) and its anticipated suc-
cess in metropolitan- and wide-area networks (MANs and
WANs), Ethernet continues to evolve to meet the increas-
ing demands of packet-switched networks. Furthermore, it
does so at low implementation cost while maintaining high
reliability and relative simplicity in installation (i.e., “plug-
n-play”), administration, and maintenance.
Although the recently ratified standard for 10-Gigabit
Ethernet (10GbE) differs from earlier Ethernet standards,
primarily with respect to operating only over fiber and only
in full-duplex mode, it remains Ethernet, and more impor-
tantly, does not obsolete current investments in network in-
frastructure. The 10GbE standard ensures interoperability
not only with respect to existing Ethernet but also other net-
working technologies such as SONET, thus paving the way
for Ethernet’s expanded use in MANs and WANs.
While 10GbE was mainly intended to allow for easy mi-
gration to higher performance levels in backbone infras-
This work was supported by the US DOE Office of Science through
LANL contract W-7405-ENG-36. This paper is also available as the fol-
lowing technical report: LA-UR 03-2713, April 2003.
tructures, we show in this paper that such performance can
also be delivered to bandwidth-hungryhost applications via
the new 10GbE network interface card (or adapter) from
Intel
R
. We first begin with an architectural overview of
the adapter in Section 2. In Sections 3 and 4, we present
the testing environments and experiments for the 10GbE
adapters, respectively. Section 5 provides results and analy-
sis. In Section 6, we examine the bottlenecks that currently
impede achieving greater performance. Section 7 compares
the 10GbE results with other high-speed interconnects. Fi-
nally, we make a few concluding remarks in Section 8.
2. Architecture of a 10GbE Adapter
The world’s first host-based 10GbE adapter, officially
known as the Intel
R
PRO/10GbE LR server adapter, intro-
duces benefits of 10GbE connectivity into LAN and SAN
environments, thereby accommodating the growing number
of large-scale systems and bandwidth-intensiveapplications
such as imaging and data mirroring. This first-generation
10GbE adapter contains a 10GbE controller that is imple-
mented in a single chip and contains both the MAC and
PHY layer functions. The controller, in turn, is optimized
for servers that use the PCI and PCI-X I/O bus backplanes.
Figure 1 provides an architectural overview of the
10GbE adapter, which consists of three main components:
82597EX 10GbE controller, 512-KB of flash memory, and
1310-nm serial optics. The 10GbE controller provides an
Ethernet interface that delivers high performance by pro-
viding direct access to all memory without using mapping
registers, minimizing interrupts and programmed I/O (PIO)
read access that are required to manage the device, and off-
loading the host CPU of simple tasks such as TCP check-
sum calculations.
As is common practice with high-performance adapters
such as Myricom’s Myrinet [2] and Quadrics’ QsNet [12],
the 10GbE adapter frees up host-CPU cycles by performing
certain tasks (in silicon) on behalf of the host CPU. In con-
trast to Myrinet and QsNet adapters, however, the 10GbE
Proceedings of IEEE Hot Interconnects: 11th Symposium on High-Performance
Interconnects; Palo Alto, CA, USA; August 2003.