![](https://csdnimg.cn/release/download_crawler_static/80298132/bg3.jpg)
Normalized Throughput
(a)
0
10
20
30
40
(b)
Throughput (million ops/s)
40-Clients
120-Clients
80-Clients
RC Write In
RC Write Out
UD Send In
UD Send Out
Number of Clients
0
0.5
1.0
Stat
ReadDir
Mknod
0 200 400 600 800
Figure 1.
(a) Scalability Issue of RDMA in Distributed File
Systems. (b) Raw Throughput of RDMA Verbs.
this issue, we rst evaluate the metadata throughput of a dis-
tributed le system named Octopus [
23
]. Octopus is designed
with light-weight software stack, so as to better exploit the
hardware performance of RDMA and NVM. For simplicity,
Octopus is congured with a single metadata server, and is
accessed with an increasing number of clients. The server
node and client nodes are connected via a 56 Gbps Mellanox
SX-1012 switch (the detailed experimental setup is described
in Section 3.6.1). As shown in Figure 1(a), when the number
of clients increases from 40 to 120, the throughput of
Stat
and
ReadDir
operations decreases signicantly (by almost
50%). We attribute this to the poor scalability of RDMA since
Octopus itself never imposes any locking overhead for those
read-oriented operations. We also notice that the through-
put of
Mknod
declines slightly by about 5%. This is because
Octopus has to do more work for update-oriented operations
and the software layer is the main bottleneck.
We further measure the raw throughput of both inbound
and outbound RDMA verbs (shown in Figure 1(b)). Among
them, inbound verbs are the number of verbs that multiple re-
mote machines (the clients) issue to one machine (the server);
outbound verbs are those that one machine issues to multiple
remote machines. Specically, 10 threads are launched at one
server to send (recv) 32-byte outbound (inbound) messages
to (from) a number of clients. We observe that the through-
put of outbound
write
drops from 20 Mops/s to 2 Mops/s as
the number of clients grows from 10 to 800. The throughput
of both inbound write and UD send is never aected.
2.3 Resource Contention in RDMA
Message Flow of RDMA.
To better understand the root
cause of the scalability issue of RDMA, we rstly cast a glance
at the message ow of RDMA
write
between two servers:
During the initialization phase, a Queue Pair (QP) is rstly
created between two servers. It consists of a send queue
and a receive queue to store the posted requests. Besides,
one or multiple memory regions are allocated and registered
with
ibv_reg_mr
, enabling them to be directly accessed by
remote CPUs. As shown in Figure 2, to post a
write
request,
the CPU of the sender initiates a verb request and sends it
MEM
CPU&Socket
core1 core2
…
NIC
5
4
MEM
CPU&Socket
core1
…
NIC
NIC&Cache
core2
Last&Level&Cache
3
2
1
NIC&Cache
Last&Level&Cache
Figure 2. Message Flow With RDMA Network.
to the local NIC with MMIO (Memory Mapped I/O) (step 1).
After the NIC is notied with the verb request, it collects the
transferred data with DMA read (step 2), and sends it out
(step 3). When the NIC at the receiver receives this message, it
transfers the data to the memory with DMA write according
to the memory address (step 4). The CPU checks the memory
region repeatedly until discovering a new message (step 5).
While RDMA bypasses the operating system with the
lightweight network stack, we nd that resource contentions
in the NIC cache, the CPU cache, and the memory signi-
cantly degrade system performance:
Cache Contention in the NIC Cache for Outbound
Verbs.
NICs mainly cache three types of information:
1
the
mapping table between the virtual and physical address of
the registered memory,
2
QP states, and
3
a work queue
elements (WQE) cache [
17
]. FaRM [
13
] uses 2 GB huge pages
while LITE [
32
] directly register physical memory so the
mapping table size is greatly reduced (i.e.,
1
). However,
when the number of connections grows, the total size of
both QP states and WQE cache will be larger than the NIC
cache size and cause cache thrashing (i.e., step 2 in Figure 2).
To quantitatively understand such issue, we collect the
hardware counters of PCIe events when testing the raw per-
formance of RC
write
(with the same conguration as in
Figure 1(b)). The results are given in Figure 3 (a). We can
nd that before the outbound RC
write
reaches its peak
performance, the corresponding PCIe read rate almost keeps
in step with the write throughput. This is because, in each
write
request, the NIC has to read the payload via DMA read
(step 2 in Figure 2). However, when the number of clients
continues to grow, the throughput of PCIe read becomes far
higher than that of the RC
write
. This is because, when the
number of connections increases, the WQEs and QP states
are more likely to be evicted out of the NIC cache by the
newly posted one, and the NIC has to read them back from
memory when processing them. Such extra eorts lead to
the degradation of overall performance. On the contrary, the
inbound
write
verbs exhibit almost constant performance
and the PCIe read rate always keeps at a low standard, be-
cause the NIC only needs to store the messages to the local
memory without modifying the cached states.
Cache Contention in the CPU Cache for Inbound
Verbs.
We can observe that the inbound
write
verbs don’t
cause the thrashing in the NIC cache. However, with DDIO
3