大规模RDMA over Commodity Ethernet部署挑战与解决方案

版权申诉

14 浏览量更新于2024-08-13 收藏 958KB PDF 举报

随着云计算和数据中心的发展，远程直接内存访问（Remote Direct Memory Access, RDMA）技术在高可靠性和低延迟场景中的应用越来越广泛。本文《RDMA over Commodity Ethernet at Scale》由来自微软的研究团队撰写，讲述了他们在过去一年半的时间里如何利用RDMA over Ethernet (RoCE v2) 技术来支持公司高度可靠且对延迟敏感的服务。作者包括Chuanxiong Guo、Haitao Wu、Zhong Deng、Gaurav Soni、Jianxi Ye、Jitendra Padhye 和 Marina Lipshteyn。文章的核心内容涵盖了以下几个关键知识点： 1. **大规模部署挑战**：为了将RoCE v2扩展到超出传统VLAN的范围，研究者们设计了一种基于区分服务代码点 (Differentiated Services Code Point, DSCP) 的优先级流量控制 (Priority Flow Control, PFC)机制，旨在应对大规模环境下的网络管理需求。这涉及到处理PFC引发的死锁问题、RDMA传输活锁，以及网络接口控制器(NIC)上的PFC暂停帧风暴问题。 2. **安全性解决方案**：针对上述安全挑战，团队开发了相应的解决方案，以确保在大规模部署中，RDMA的运行能够保持稳定，并防止可能产生的系统不稳定因素。 3. **监控与管理系统**：构建了完整的监控和管理系统，用于实时检测和管理RoCE v2的性能，确保其按预期工作。这对于保障服务质量至关重要。 4. **性能优化**：通过使用RDMA，论文强调了其在数据中心内通信中的优势，如低延迟、低CPU占用率和高吞吐量，从而可以替代传统的TCP协议，尤其是在处理大量数据传输时。 5. **结论与展望**：研究者们的经验表明，虽然在大规模应用RDMA时会遇到一些挑战，但通过精心设计的机制和技术，这些问题都是可以解决的。这为未来在数据中心内部署和优化RDMA技术提供了宝贵的实践经验。本文不仅分享了微软在实施RDMA over Ethernet at scale过程中的实战经验和教训，还探讨了如何通过技术创新解决实际部署中的问题，推动了高性能网络技术在数据中心环境中的广泛应用和发展。

it. During this time, the upstream port will continue to

transmit packets. Thus, the ingress port must reserve

buﬀer space for each priority to absorb packets that

arrive during this “gray period”. This reserved buﬀer is

called headroom. The size of the headroom is decided by

the MTU size, the PFC reaction time of the egress port,

and most importantly, the propagation delay between

the sender and the receiver.

The propagation delay is determined by the distance

between the sender and the receiver. In our network,

this can be as large as 300 meters. Given that our ToR

and Leaf switches have shallow buﬀers (9MB or 12MB),

we can only reserve enough headroom for two lossless

traﬃc classes even though the switches support eight

traﬃc classes. We use one lossless class for real-time

traﬃc and the other for bulk data transfer.

Need for congestion control: PFC works hop by

hop. There may be several hops from the source server

to the destination server. PFC pause frames propagate

from the congestion point back to the source if there is

persistent network congestion. This can cause problems

like unfairness and victim ﬂow [42].

In order to reduce this collateral damage, ﬂow based

congestion control mechanisms including QCN [13], DC-

QCN [42] and TIMELY [27] have been introduced. We

use DCQCN, which uses ECN for congestion notiﬁca-

tion, in our network. We chose DCQCN because it di-

rectly reacts to the queue lengths at the intermediate

switches and ECN is well supported by all the switches

we use. Small queue lengths reduce the PFC generation

and propagation probability.

Though DCQCN helps reduce the number of PFC

pause frames, it is PFC that protects packets from being

dropped as the last defense. PFC poses several safety is-

sues which are the primary focus of this paper and which

we will discuss in Section 4. We believe the lessons we

have learned in this paper apply to the networks using

TIMELY as well.

Coexistence of RDMA and TCP: In this paper,

RDMA is designed for intra-DC communications. TCP

is still needed for inter-DC communications and legacy

applications. We use a diﬀerent traﬃc class (which is

not lossless), with reserved bandwidth, for TCP. Diﬀer-

ent traﬃc classes isolate TCP and RDMA traﬃc from

each other.

3. DSCP-BASED PFC

In this section we examine the issues faced by the

original VLAN-based PFC and present our DSCP-based

PFC solution. VLAN-based PFC carries packet prior-

ity in the VLAN tag, which also contains VLAN ID.

The coupling of packet priority and VLAN ID created

two serious problems in our deployment, leading us to

develop a DSCP-based PFC solution.

Figure 3(a) shows the packet formats of the PFC

pause frame and data packets in the original VLAN-

based PFC. The pause frame is a layer-2 frame, and

(a) VLAN-based PFC.

(b) DSCP-based PFC.

Figure 3: The packet formats of VLAN-based PFC and

DSCP-based PFC. Note that the PFC pause frame for-

mat is the same in both Figure 3(a) and Figure 3(b).

does not have a VLAN tag. The VLAN tag for the

data packet has four parts: TPID which is ﬁxed to

0x8100, DEI (Drop Eligible Indicator), PCP (Priority

Code Point) which is used to carry packet priority, and

VID (VLAN identiﬁer) which carries the VLAN ID of

the packet.

For our purpose, although we need only PCP, VID

and PCP cannot be separated. Thus, to support PFC,

we have to conﬁgure VLAN at both the server and the

switch side. In order for the switch ports to support

VLAN, we need to put the server facing switch ports

into trunk mode (which supports VLAN tagged pack-

ets) instead of access mode (which sends and receives

untagged packets). The basic PFC functionality works

with this conﬁguration, but it leads to two problems.

First, the switch trunk mode has an undesirable inter-

action with our OS provisioning service. OS provision-

ing is a fundamental service which needs to run when

the server OS needs to be installed or upgraded, and

when the servers need to be provisioned or repaired.

For data centers at our scale, OS provisioning has to

be done automatically. We use PXE (Preboot eXecu-

tion Environment) boot to install OS from the network.

When a server goes through PXE boot, its NIC does

not have VLAN conﬁguration and as a result cannot

send or receive packets with VLAN tags. But since

the server facing switch ports are conﬁgured with trunk

mode, these ports can only send packets with VLAN

tag. Hence the PXE boot communication between the

剩余13页未读，继续阅读

工控老马

粉丝: 683
资源: 2561

大规模RDMA over Commodity Ethernet部署挑战与解决方案

Implementation & Comparison of RDMA Over Ethernet.pdf

rdma优势及Roce简介.pdf

浅析RDMA网络下MMU水线设置.pdf

Ethernet RDMA Technologies.pdf

NFS-RDMA安装配置[文].pdf

Towards a Lightweight RDMA Para-Virtualization for HPC.pdf

RDMA_Aware_Programming.pdf

RDMA原理分析及实现解析.zip

VMware Paravirtual RDMA for High Performance Computing.pdf

Xilinx Embedded RDMA Enabled NIC v3.0.pdf

最新资源