ScaleRPC：一种基于可靠连接的可扩展RDMA远程过程调用

需积分: 9 49 浏览量更新于2024-08-05 收藏 1.36MB PDF 举报

"基于可靠连接和高效资源共享的可扩展RDMA RPC" 在分布式系统中，远程直接内存访问（RDMA）因其极低的延迟和高带宽特性而备受青睐。然而，当通过可靠的连接（RC）向越来越多的目标传输数据时，RDMA的性能扩展性不佳，且会遭遇性能退化的问题。这个问题的根本原因在于网络接口卡（NIC）缓存、CPU缓存以及每个服务器的内存中的资源竞争。针对这一挑战，陈业明、吕悠游和舒继武等人提出了一种名为ScaleRPC的高效RPC（远程过程调用）原语，它利用可靠的连接上的单边RDMA操作来实现可扩展的性能。ScaleRPC旨在有效缓解资源竞争，从而提升系统的整体效率。 1) 连接分组（Connection Grouping）：ScaleRPC引入了连接分组的概念，将网络连接组织成多个组，以平衡NIC缓存的饱和度和thrashing（缓存替换频繁导致的性能下降）。通过这种方式，可以确保每个连接组都能获得相对均衡的资源，避免单一连接过度占用资源，提高整个系统的吞吐量。 2) 虚拟映射（Virtualized Mapping）：为了减少CPU缓存未命中的情况并优化内存利用率，ScaleRPC实现了虚拟映射功能，允许不同的连接组共享同一个消息池。这样，来自不同连接的数据请求可以更有效地复用缓存空间，降低内存访问延迟，进一步提高系统性能。 3) 高效资源调度：ScaleRPC还可能包含了高效的资源调度策略，以动态调整连接组的资源分配，适应不断变化的工作负载。这可能包括基于优先级的调度或基于预测的资源预分配，以最大化资源利用并最小化延迟。 4) 异步处理：考虑到RDMA的单边操作特性，ScaleRPC可能利用异步处理机制，使得发送端可以在等待响应的同时执行其他任务，提高了并发处理能力，降低了系统的整体等待时间。 5) 故障恢复和可靠性：在可靠的连接上使用RDMA，ScaleRPC必须具备有效的错误检测和恢复机制，以保证服务的高可用性和数据的完整性。这可能包括冗余连接、心跳检测和重传策略等。 ScaleRPC是一种创新的RPC解决方案，它通过连接分组和虚拟映射等技术解决了RDMA在可靠连接下的扩展性问题，同时通过高效资源管理和故障恢复策略保证了系统的稳定性和可靠性。这一工作对构建大规模分布式系统提供了新的思路，有望推动RDMA技术在云存储、大数据处理和高性能计算等领域的应用。

Normalized Throughput

(a)

(b)

Throughput (million ops/s)

40-Clients

120-Clients

80-Clients

RC Write In

RC Write Out

UD Send In

UD Send Out

Number of Clients

0.5

1.0

Stat

ReadDir

Mknod

0 200 400 600 800

Figure 1.

(a) Scalability Issue of RDMA in Distributed File

Systems. (b) Raw Throughput of RDMA Verbs.

this issue, we rst evaluate the metadata throughput of a dis-

tributed le system named Octopus [

]. Octopus is designed

with light-weight software stack, so as to better exploit the

hardware performance of RDMA and NVM. For simplicity,

Octopus is congured with a single metadata server, and is

accessed with an increasing number of clients. The server

node and client nodes are connected via a 56 Gbps Mellanox

SX-1012 switch (the detailed experimental setup is described

in Section 3.6.1). As shown in Figure 1(a), when the number

of clients increases from 40 to 120, the throughput of

Stat

and

ReadDir

operations decreases signicantly (by almost

50%). We attribute this to the poor scalability of RDMA since

Octopus itself never imposes any locking overhead for those

read-oriented operations. We also notice that the through-

put of

Mknod

declines slightly by about 5%. This is because

Octopus has to do more work for update-oriented operations

and the software layer is the main bottleneck.

We further measure the raw throughput of both inbound

and outbound RDMA verbs (shown in Figure 1(b)). Among

them, inbound verbs are the number of verbs that multiple re-

mote machines (the clients) issue to one machine (the server);

outbound verbs are those that one machine issues to multiple

remote machines. Specically, 10 threads are launched at one

server to send (recv) 32-byte outbound (inbound) messages

to (from) a number of clients. We observe that the through-

put of outbound

write

drops from 20 Mops/s to 2 Mops/s as

the number of clients grows from 10 to 800. The throughput

of both inbound write and UD send is never aected.

2.3 Resource Contention in RDMA

Message Flow of RDMA.

To better understand the root

cause of the scalability issue of RDMA, we rstly cast a glance

at the message ow of RDMA

write

between two servers:

During the initialization phase, a Queue Pair (QP) is rstly

created between two servers. It consists of a send queue

and a receive queue to store the posted requests. Besides,

one or multiple memory regions are allocated and registered

with

ibv_reg_mr

, enabling them to be directly accessed by

remote CPUs. As shown in Figure 2, to post a

write

request,

the CPU of the sender initiates a verb request and sends it

MEM

CPU&Socket

core1 core2

…

NIC

MEM

CPU&Socket

core1

…

NIC

NIC&Cache

core2

Last&Level&Cache

NIC&Cache

Last&Level&Cache

Figure 2. Message Flow With RDMA Network.

to the local NIC with MMIO (Memory Mapped I/O) (step 1).

After the NIC is notied with the verb request, it collects the

transferred data with DMA read (step 2), and sends it out

(step 3). When the NIC at the receiver receives this message, it

transfers the data to the memory with DMA write according

to the memory address (step 4). The CPU checks the memory

region repeatedly until discovering a new message (step 5).

While RDMA bypasses the operating system with the

lightweight network stack, we nd that resource contentions

in the NIC cache, the CPU cache, and the memory signi-

cantly degrade system performance:

Cache Contention in the NIC Cache for Outbound

Verbs.

NICs mainly cache three types of information:



the

mapping table between the virtual and physical address of

the registered memory,



QP states, and



a work queue

elements (WQE) cache [

]. FaRM [

] uses 2 GB huge pages

while LITE [

] directly register physical memory so the

mapping table size is greatly reduced (i.e.,



). However,

when the number of connections grows, the total size of

both QP states and WQE cache will be larger than the NIC

cache size and cause cache thrashing (i.e., step 2 in Figure 2).

To quantitatively understand such issue, we collect the

hardware counters of PCIe events when testing the raw per-

formance of RC

write

(with the same conguration as in

Figure 1(b)). The results are given in Figure 3 (a). We can

nd that before the outbound RC

write

reaches its peak

performance, the corresponding PCIe read rate almost keeps

in step with the write throughput. This is because, in each

write

request, the NIC has to read the payload via DMA read

(step 2 in Figure 2). However, when the number of clients

continues to grow, the throughput of PCIe read becomes far

higher than that of the RC

write

. This is because, when the

number of connections increases, the WQEs and QP states

are more likely to be evicted out of the NIC cache by the

newly posted one, and the NIC has to read them back from

memory when processing them. Such extra eorts lead to

the degradation of overall performance. On the contrary, the

inbound

write

verbs exhibit almost constant performance

and the PCIe read rate always keeps at a low standard, be-

cause the NIC only needs to store the messages to the local

memory without modifying the cached states.

Cache Contention in the CPU Cache for Inbound

Verbs.

We can observe that the inbound

write

verbs don’t

cause the thrashing in the NIC cache. However, with DDIO

剩余13页未读，继续阅读

bandaoyu

粉丝: 17w+
资源: 66

ScaleRPC：一种基于可靠连接的可扩展RDMA远程过程调用

StarR：打破 RDMA 的可扩展性限制

RDMA编程实例源码（rdma_cm API）

什么是gRPC和RDMA

RDMA ROcE和Iwarp区别

rdma原理分析和技术实现 pdf

RDMA和DMAS区别

rdma program 指南

RDMA学习方法，RDMA的拥塞算法现状及演进方向

使用GPUDirect RDMA支持rdma

gpu direct rdma 支持

最新资源