Mellanox RDMA-Aware Networking编程用户手册v1.3

需积分: 0 65 浏览量更新于2024-06-25 收藏 2.22MB PDF 举报

本资源是一份名为《RDMA-Aware Programming 用户手册》的PDF文档，由Mellanox Technologies发布，版本为Rev 1.3。Mellanox是一家专注于高速网络技术和高性能计算解决方案的公司，其产品和服务在数据中心、云计算和HPC（High Performance Computing）领域享有盛誉。该用户手册的主题围绕RDMA（Remote Direct Memory Access）技术展开，这是一种在计算机网络中实现高效数据传输的技术，允许应用程序直接访问远程内存，无需经过CPU干预，显著提高了数据传输速度和吞吐量，对于大规模并行计算环境尤其重要。RDMA通常与InfiniBand（IB）网络标准紧密关联，这种标准被设计用于提供低延迟、高带宽和高可靠性的连接。手册内容涵盖了如何在编程中充分利用RDMA特性，包括如何设计和优化应用程序以利用Mellanox的硬件如ConnectX、InfiniBand和InfiniScale等高性能网络接口卡。此外，手册还可能介绍了Mellanox提供的软件工具，如MLNX-OS（Mellanox Linux Network Operating System），以及与RDMA相关的概念和技术细节，如Connect-IB（Connect to InfiniBand）、FabricIT（Fabric Interface Technology）和统一的 Fabric Manager（Unified Fabric Manager），这些都能帮助开发人员更好地理解和利用RDMA技术来提升系统的性能和效率。用户在阅读此手册时，可以期待学习到以下关键知识点： 1. **RDMA原理和优势**：理解RDMA的基本原理，包括其工作方式、性能提升和在分布式计算中的应用价值。 2. **API和编程接口**：学习如何通过Mellanox提供的API或SDK（Software Development Kit）调用RDMA功能，以及如何编写高效的RDMA程序。 3. **优化策略**：如何根据应用程序需求调整网络参数，比如减少内存复制操作，提高内存带宽利用率。 4. **故障检测和恢复**：关于如何处理和诊断RDMA相关问题，以及如何设计健壮的容错机制。 5. **案例研究和示例**：手册可能包含实际项目中的应用案例，帮助读者理解和实践RDMA技术。这份用户手册是针对那些希望深入了解和在实践中使用RDMA技术的IT专业人士的重要参考资料，无论是系统架构师、网络工程师还是高级开发者，都将从中受益匪浅。

RDMA Architecture OverviewRev 1.3

Mellanox Technologies

address translation mechanism under the control of the operating system which allows an applica-

tion to access the HCA directly. The same address translation mechanism is the means by which an

HCA accesses memory on behalf of a user level application. The application refers to virtual

addresses while the HCA has the ability to translate these addresses into physical addresses in

order to affect the actual message transfer.

Range Extenders

InfiniBand range extension is accomplished by encapsulating the InfiniBand traffic onto the WAN

link and extending sufficient buffer credits to ensure full bandwidth across the WAN.

Subnet Manager

The InfiniBand subnet manager assigns Local Identifiers (LIDs) to each port connected to the

InfiniBand fabric and develops a routing table based on the assigned LIDs. The IB Subnet Man-

ager is a concept of Software Defined Networking (SDN) which eliminates the interconnect com-

plexity and enables the creation of very large scale compute and storage infrastructures.

Switches

IB switches are conceptually similar to standard networking switches but are designed to meet IB

performance requirements. They implement flow control of the IB Link Layer to prevent packet

dropping, and to support congestion avoidance and adaptive routing capabilities, and advanced

Quality of Service. Many switches include a Subnet Manager. At least one Subnet Manager is

required to configure an IB fabric.

1.5 Support for Existing Applications and ULPs

IP applications are enabled to run over an InfiniBand fabric using IP over IB (IPoIB) or Ethernet

over IB (EoIB) or RDS ULPs. Storage applications are supported via iSER, SRP, RDS, NFS, ZFS,

SMB and others. MPI and Network Direct are all supported ULPs as well, but are outside the

scope of this document.

RDMA Aware Networks Programming User Manual Rev 1.3

Mellanox Technologies

3 Overview

In order to perform RDMA operations, establishment of a connection to the remote host, as well as

appropriate permissions need to be set up first. The mechanism for accomplishing this is the

Queue Pair (QP). For those familiar with a standard IP stack, a QP is roughly equivalent to a

socket. The QP needs to be initialized on both sides of the connection. Communication Manager

(CM) can be used to exchange information about the QP prior to actual QP setup.

Once a QP is established, the verbs API can be used to perform RDMA reads, RDMA writes, and

atomic operations. Serialized send/receive operations, which are similar to socket reads/writes, can

be performed as well.

3.1 Available Communication Operations

3.1.1 Send/Send With Immediate

The send operation allows you to send data to a remote QP’s receive queue. The receiver must

have previously posted a receive buffer to receive the data. The sender does not have any control

over where the data will reside in the remote host.

Optionally, an immediate 4 byte value may be transmitted with the data buffer. This immediate

value is presented to the receiver as part of the receive notification, and is not contained in the data

buffer.

3.1.2 Receive

This is the corresponding operation to a send operation. The receiving host is notified that a data

buffer has been received, possibly with an inline immediate value. The receiving application is

responsible for receive buffer maintenance and posting.

3.1.3 RDMA Read

A section of memory is read from the remote host. The caller specifies the remote virtual address

as well as a local memory address to be copied to. Prior to performing RDMA operations, the

remote host must provide appropriate permissions to access its memory. Once these permissions

are set, RDMA read operations are conducted with no notification whatsoever to the remote host.

For both RDMA read and write, the remote side isn't aware that this operation being done (other

than the preparation of the permissions and resources).

3.1.4 RDMA Write / RDMA Write With Immediate

Similar to RDMA read, but the data is written to the remote host. RDMA write operations are per-

formed with no notification to the remote host. RDMA write with immediate operations, however,

do notify the remote host of the immediate value.

3.1.5 Atomic Fetch and Add / Atomic Compare and Swap

These are atomic extensions to the RDMA operations.

剩余246页未读，继续阅读

109702008

粉丝: 1w+
资源: 146

Mellanox RDMA-Aware Networking编程用户手册v1.3

RDMA_Aware_Programming_user_manual.pdf

RDMA_Aware_Programming.pdf

RDMA_Aware_Programming_user_manual.pdf迈络斯RDMA 编程指南

rdma-core-devel-17.2-3.el7.x86_64

rdma-core-22.1-3.el7.x86_64

rdma-core依赖下载

Designing Distributed Tree-based Index Structures for Fast RDMA-capable Networks

grep: /usr/src/ofa_kernel/aarch64/include/rdma/ib_verbs.h: No such file or directory

如何为GPU RDMA编写代码

infiniband的rdma技术如何使用，在centos7上如果安装使用

最新资源