迈络斯RDMA编程指南：用户手册

需积分: 48 75 浏览量更新于2024-07-16 2 收藏 1.89MB PDF 举报

"RDMA_Aware_Programming_user_manual.pdf 是迈络斯（Mellanox）公司关于RDMA（远程直接内存访问）编程的一份指南，涵盖了C/C++语言的编程说明和实例，旨在帮助开发者理解和掌握RDMA技术在高性能计算和网络通信中的应用。" 在RDMA（Remote Direct Memory Access）技术中，应用程序可以直接读写远程机器的内存，而无需通过操作系统内核，从而减少了CPU的负担，提高了数据传输速度和系统效率。Mellanox是该领域的领导者，提供了一系列基于RDMA技术的硬件和软件解决方案。本用户手册可能包括以下几个主要知识点： 1. **RDMA基础知识**：解释了RDMA的工作原理，包括RDMA的两种模式——Read和Write，以及如何通过InfiniBand或Ethernet网络实现低延迟、高带宽的数据传输。 2. **Mellanox RDMA技术**：介绍了Mellanox的ConnectX系列网卡，它们支持RDMA，并且提供了诸如CORE-Direct、InfiniBridge等专有技术，以优化性能和降低延迟。 3. **RDMA编程接口**：详细阐述了如何在C/C++中使用RDMA编程接口，包括初始化、建立连接、数据传输以及错误处理等步骤。 4. **示例代码**：提供了实际的编程示例，让开发者能够直观地理解如何在自己的应用中集成RDMA功能。 5. **性能优化**：讨论了如何最大化利用RDMA的性能，包括缓存管理、工作队列设置、数据包大小优化等。 6. **故障排查与调试**：介绍了在遇到问题时如何进行诊断，以及使用Mellanox的工具如UFM（Unified Fabric Manager）进行网络管理和监控。 7. **安全性与可靠性**：讲解了在实现RDMA通信时如何保证数据的安全性和系统的稳定性。 8. **应用领域**：强调了RDMA技术在高性能计算、大数据、云计算、存储等领域的重要应用。这份手册对于希望深入理解和使用RDMA技术的开发者来说，是一份非常宝贵的参考资料，它不仅提供了理论知识，还结合实践案例帮助读者掌握实际操作技能。然而，需要注意的是，产品和文档都是“按原样”提供的，未经过全面的生产环境验证，因此在实际部署时，开发者还需要根据自身环境进行充分的测试和调整。

RDMA Architecture OverviewRev 1.7

Mellanox Technologies

The most important point is that every application has direct access to the virtual memory of

devices in the fabric. This means that applications do not need to make requests to an operating

system to transfer messages. Contrast this with the traditional network environment where the

shared network resources are owned by the operating system and cannot be accessed by a user

application. Thus, an application must rely on the involvement of the operating system to move

data from the application's virtual buffer space, through the network stack and out onto the wire.

Similarly, at the other end, an application must rely on the operating system to retrieve the data

on the wire on its behalf and place it in its virtual buffer space.

TCP/IP/Ethernet is a byte-stream oriented transport for passing bytes of information between

sockets applications. TCP/IP is lossy by design but implements a reliability scheme using the

Transmission Control Protocol (TCP). TCP/IP requires Operating System (OS) intervention for

every operation which includes buffer copying on both ends of the wire. In a byte stream-ori-

Rev 1.7

Mellanox Technologies

ented network, the idea of a message boundary is lost. When an application wants to send a

packet, the OS places the bytes into an anonymous buffer in main memory belonging to the oper-

ating system and when the byte transfer is complete, the OS copies the data in its buffer into the

receive buffer of the application. This process is repeated each time a packet arrives until the

entire byte stream is received. TCP is responsible for retransmitting any lost packets due to con-

gestion.

In IB, a complete message is delivered directly to an application. Once an application has

requested transport of an RDMA Read or Write, the IB hardware segments the outbound mes-

sage as needed into packets whose size is determined by the fabric path maximum transfer unit.

These packets are transmitted through the IB network and delivered directly into the receiving

application's virtual buffer where they are re-assembled into a complete message. The receiving

application is notified once the entire message has been received. Thus neither the sending nor

the receiving application is involved until the entire message is delivered into the receiving appli-

cation's buffer.

1.5 Key Components

These are being presented only in the context of the advantages of deploying IB and RoCE. We

do not discuss cables and connectors.

Host Channel Adapter

HCAs provide the point at which an IB end node (for example, a server) connects to an IB net-

work. These are the equivalent of the Ethernet (NIC) card but they do much more. HCAs provide

address translation mechanism under the control of the operating system which allows an appli-

cation to access the HCA directly. The same address translation mechanism is the means by

which an HCA accesses memory on behalf of a user level application. The application refers to

virtual addresses while the HCA has the ability to translate these addresses into physical

addresses in order to affect the actual message transfer.

Range Extenders

InfiniBand range extension is accomplished by encapsulating the InfiniBand traffic onto the

WAN link and extending sufficient buffer credits to ensure full bandwidth across the WAN.

Subnet Manager

The InfiniBand subnet manager assigns Local Identifiers (LIDs) to each port connected to the

InfiniBand fabric and develops a routing table based on the assigned LIDs. The IB Subnet Man-

ager is a concept of Software Defined Networking (SDN) which eliminates the interconnect

complexity and enables the creation of very large scale compute and storage infrastructures.

Switches

IB switches are conceptually similar to standard networking switches but are designed to meet IB

performance requirements. They implement flow control of the IB Link Layer to prevent packet

dropping, and to support congestion avoidance and adaptive routing capabilities, and advanced

Quality of Service. Many switches include a Subnet Manager. At least one Subnet Manager is

required to configure an IB fabric.

1.6 Support for Existing Applications and ULPs

IP applications are enabled to run over an InfiniBand fabric using IP over IB (IPoIB) or Ethernet

over IB (EoIB) or RDS ULPs. Storage applications are supported via iSER, SRP, RDS, NFS,

Rev 1.7

Mellanox Technologies

2 RDMA-Aware Programming Overview

The VPI architecture permits direct user mode access to the hardware. Mellanox provides a

dynamically loaded library, creating access to the hardware via the verbs API. This document

contains verbs and their related inputs, outputs, descriptions, and functionality as exposed

through the operating system programming interface.

Note: This programming manual and its verbs are valid only for user space. See header files

for the kernel space verbs.

Programming with verbs allows for customizing and optimizing the RDMA-Aware network.

This customizing and optimizing should be done only by programmers with advanced knowl-

edge and experience in the VPI systems.

In order to perform RDMA operations, establishment of a connection to the remote host, as well

as appropriate permissions need to be set up first. The mechanism for accomplishing this is the

Queue Pair (QP). For those familiar with a standard IP stack, a QP is roughly equivalent to a

socket. The QP needs to be initialized on both sides of the connection. Communication Manager

(CM) can be used to exchange information about the QP prior to actual QP setup.

Once a QP is established, the verbs API can be used to perform RDMA reads, RDMA writes, and

atomic operations. Serialized send/receive operations, which are similar to socket reads/writes,

can be performed as well.

2.1 Available Communication Operations

2.1.1 Send/Send With Immediate

The send operation allows you to send data to a remote QP’s receive queue. The receiver must

have previously posted a receive buffer to receive the data. The sender does not have any control

over where the data will reside in the remote host.

Optionally, an immediate 4 byte value may be transmitted with the data buffer. This immediate

value is presented to the receiver as part of the receive notification, and is not contained in the

data buffer.

2.1.2 Receive

This is the corresponding operation to a send operation. The receiving host is notified that a data

buffer has been received, possibly with an inline immediate value. The receiving application is

responsible for receive buffer maintenance and posting.

2.1.3 RDMA Read

A section of memory is read from the remote host. The caller specifies the remote virtual address

as well as a local memory address to be copied to. Prior to performing RDMA operations, the

remote host must provide appropriate permissions to access its memory. Once these permissions

are set, RDMA read operations are conducted with no notification whatsoever to the remote host.

For both RDMA read and write, the remote side isn't aware that this operation being done (other

than the preparation of the permissions and resources).

剩余215页未读，继续阅读

杰特JET

粉丝: 32
资源: 3

迈络斯RDMA编程指南：用户手册

RDMA编程用户手册-官方中文版.pdf

RDMA编程手册11111

RDMA编程手册 V1.4中文前2章

RDMA_Aware_Programming_user_manual.pdf

RDMA_Aware_Programming_user_manual.zip

RDMA_Aware_Programming_user_manual.rar

RDMA-Aware-Programming-user-manual-1.pdf

RDMA Aware Networks Programming User Manual

RDMA编程用户手册

RDMA编程指南：架构与API详解

最新资源