Mellanox InfiniBand RDMA编程手册：Rev1.7技术详解

需积分: 36 156 浏览量更新于2024-07-19 2 收藏 2.26MB PDF 举报

《RDMA-aware网络编程用户手册》是Mellanox Technologies提供的全面指南，专为理解和开发InfiniBand Remote Direct Memory Access (RDMA)技术而设计。该文档适用于那些希望在高性能计算(HPC)、数据中心和云计算环境中利用RDMA特性提升网络性能和效率的开发者。RDMA是一种低延迟、带宽密集型通信协议，它允许数据直接在远程节点的内存之间传输，无需CPU介入，从而减少数据包处理开销。手册Rev1.7包含了关于以下关键内容： 1. **InfiniBand基础知识**：介绍InfiniBand架构、体系结构和基本概念，包括其在现代网络中的角色以及如何通过InfiniBand设备（如ConnectX系列）实现高速、无损的数据传输。 2. **RDMA编程接口**：讲解如何使用特定编程语言（如C/C++、Java等）来编写RDMA应用程序，包括使用InfiniBand SDK（Software Development Kit）提供的API，如Verbs (verbs) API，进行内存直接映射操作和完成路径（Completion Path）管理。 3. **网络编程示例**：提供实用的代码片段和案例研究，展示如何在实际场景中应用RDMA技术，如大规模数据传输、分布式系统通信和高并发环境下的优化。 4. **硬件兼容性与驱动管理**：列出支持RDMA的硬件设备清单，并解释如何管理和配置InfiniBand网络控制器和适配器，确保最佳性能。 5. **故障排查与调试**：对于遇到的问题和常见错误，手册提供了故障诊断和解决策略，帮助开发者快速定位和修复问题。 6. **安全与认证**：强调了在RDMA网络中保护数据隐私和完整性的重要性，以及Mellanox提供的安全特性，如Unbreakable-Link和Secure Data Center Bridging (SDBC)。 7. **版权和许可信息**：最后，手册强调了所有内容的版权归属，以及对Mellanox和其他供应商商标的尊重。此手册是为那些希望通过RDMA技术改进网络性能，尤其是追求低延迟、高带宽和可扩展性的工程师、系统管理员和开发者设计的权威参考资料。阅读并掌握其中的内容，将有助于优化系统架构，提高数据中心和HPC环境的性能水平。

RDMA Architecture OverviewRev 1.7

Mellanox Technologies

The most important point is that every application has direct access to the virtual memory of

devices in the fabric. This means that applications do not need to make requests to an operating

system to transfer messages. Contrast this with the traditional network environment where the

shared network resources are owned by the operating system and cannot be accessed by a user

application. Thus, an application must rely on the involvement of the operating system to move

data from the application's virtual buffer space, through the network stack and out onto the wire.

Similarly, at the other end, an application must rely on the operating system to retrieve the data

on the wire on its behalf and place it in its virtual buffer space.

TCP/IP/Ethernet is a byte-stream oriented transport for passing bytes of information between

sockets applications. TCP/IP is lossy by design but implements a reliability scheme using the

Transmission Control Protocol (TCP). TCP/IP requires Operating System (OS) intervention for

every operation which includes buffer copying on both ends of the wire. In a byte stream-ori-

Rev 1.7

Mellanox Technologies

ented network, the idea of a message boundary is lost. When an application wants to send a

packet, the OS places the bytes into an anonymous buffer in main memory belonging to the oper-

ating system and when the byte transfer is complete, the OS copies the data in its buffer into the

receive buffer of the application. This process is repeated each time a packet arrives until the

entire byte stream is received. TCP is responsible for retransmitting any lost packets due to con-

gestion.

In IB, a complete message is delivered directly to an application. Once an application has

requested transport of an RDMA Read or Write, the IB hardware segments the outbound mes-

sage as needed into packets whose size is determined by the fabric path maximum transfer unit.

These packets are transmitted through the IB network and delivered directly into the receiving

application's virtual buffer where they are re-assembled into a complete message. The receiving

application is notified once the entire message has been received. Thus neither the sending nor

the receiving application is involved until the entire message is delivered into the receiving appli-

cation's buffer.

1.5 Key Components

These are being presented only in the context of the advantages of deploying IB and RoCE. We

do not discuss cables and connectors.

Host Channel Adapter

HCAs provide the point at which an IB end node (for example, a server) connects to an IB net-

work. These are the equivalent of the Ethernet (NIC) card but they do much more. HCAs provide

address translation mechanism under the control of the operating system which allows an appli-

cation to access the HCA directly. The same address translation mechanism is the means by

which an HCA accesses memory on behalf of a user level application. The application refers to

virtual addresses while the HCA has the ability to translate these addresses into physical

addresses in order to affect the actual message transfer.

Range Extenders

InfiniBand range extension is accomplished by encapsulating the InfiniBand traffic onto the

WAN link and extending sufficient buffer credits to ensure full bandwidth across the WAN.

Subnet Manager

The InfiniBand subnet manager assigns Local Identifiers (LIDs) to each port connected to the

InfiniBand fabric and develops a routing table based on the assigned LIDs. The IB Subnet Man-

ager is a concept of Software Defined Networking (SDN) which eliminates the interconnect

complexity and enables the creation of very large scale compute and storage infrastructures.

Switches

IB switches are conceptually similar to standard networking switches but are designed to meet IB

performance requirements. They implement flow control of the IB Link Layer to prevent packet

dropping, and to support congestion avoidance and adaptive routing capabilities, and advanced

Quality of Service. Many switches include a Subnet Manager. At least one Subnet Manager is

required to configure an IB fabric.

1.6 Support for Existing Applications and ULPs

IP applications are enabled to run over an InfiniBand fabric using IP over IB (IPoIB) or Ethernet

over IB (EoIB) or RDS ULPs. Storage applications are supported via iSER, SRP, RDS, NFS,

Rev 1.7

Mellanox Technologies

2 RDMA-Aware Programming Overview

The VPI architecture permits direct user mode access to the hardware. Mellanox provides a

dynamically loaded library, creating access to the hardware via the verbs API. This document

contains verbs and their related inputs, outputs, descriptions, and functionality as exposed

through the operating system programming interface.

Note: This programming manual and its verbs are valid only for user space. See header files

for the kernel space verbs.

Programming with verbs allows for customizing and optimizing the RDMA-Aware network.

This customizing and optimizing should be done only by programmers with advanced knowl-

edge and experience in the VPI systems.

In order to perform RDMA operations, establishment of a connection to the remote host, as well

as appropriate permissions need to be set up first. The mechanism for accomplishing this is the

Queue Pair (QP). For those familiar with a standard IP stack, a QP is roughly equivalent to a

socket. The QP needs to be initialized on both sides of the connection. Communication Manager

(CM) can be used to exchange information about the QP prior to actual QP setup.

Once a QP is established, the verbs API can be used to perform RDMA reads, RDMA writes, and

atomic operations. Serialized send/receive operations, which are similar to socket reads/writes,

can be performed as well.

2.1 Available Communication Operations

2.1.1 Send/Send With Immediate

The send operation allows you to send data to a remote QP’s receive queue. The receiver must

have previously posted a receive buffer to receive the data. The sender does not have any control

over where the data will reside in the remote host.

Optionally, an immediate 4 byte value may be transmitted with the data buffer. This immediate

value is presented to the receiver as part of the receive notification, and is not contained in the

data buffer.

2.1.2 Receive

This is the corresponding operation to a send operation. The receiving host is notified that a data

buffer has been received, possibly with an inline immediate value. The receiving application is

responsible for receive buffer maintenance and posting.

2.1.3 RDMA Read

A section of memory is read from the remote host. The caller specifies the remote virtual address

as well as a local memory address to be copied to. Prior to performing RDMA operations, the

remote host must provide appropriate permissions to access its memory. Once these permissions

are set, RDMA read operations are conducted with no notification whatsoever to the remote host.

For both RDMA read and write, the remote side isn't aware that this operation being done (other

than the preparation of the permissions and resources).

剩余215页未读，继续阅读

Latterrs

粉丝: 10
资源: 15

Mellanox InfiniBand RDMA编程手册：Rev1.7技术详解

RDMA编程手册 V1.4中文前2章

RDMA编程用户手册-官方中文版.pdf

RDMA_Aware_Programming_user_manual.rar

Designing Distributed Tree-based Index Structures for Fast RDMA-capable Networks

使用GPUDirect RDMA支持rdma

如何为GPU RDMA编写代码

RDMA在CUDA中的应用

rdma在Linux下如何设置

如何确定rdma已安装

gpu direct rdma 支持

最新资源