Rev 1.7
Mellanox Technologies
17
ented network, the idea of a message boundary is lost. When an application wants to send a
packet, the OS places the bytes into an anonymous buffer in main memory belonging to the oper-
ating system and when the byte transfer is complete, the OS copies the data in its buffer into the
receive buffer of the application. This process is repeated each time a packet arrives until the
entire byte stream is received. TCP is responsible for retransmitting any lost packets due to con-
gestion.
In IB, a complete message is delivered directly to an application. Once an application has
requested transport of an RDMA Read or Write, the IB hardware segments the outbound mes-
sage as needed into packets whose size is determined by the fabric path maximum transfer unit.
These packets are transmitted through the IB network and delivered directly into the receiving
application's virtual buffer where they are re-assembled into a complete message. The receiving
application is notified once the entire message has been received. Thus neither the sending nor
the receiving application is involved until the entire message is delivered into the receiving appli-
cation's buffer.
1.5 Key Components
These are being presented only in the context of the advantages of deploying IB and RoCE. We
do not discuss cables and connectors.
Host Channel Adapter
HCAs provide the point at which an IB end node (for example, a server) connects to an IB net-
work. These are the equivalent of the Ethernet (NIC) card but they do much more. HCAs provide
address translation mechanism under the control of the operating system which allows an appli-
cation to access the HCA directly. The same address translation mechanism is the means by
which an HCA accesses memory on behalf of a user level application. The application refers to
virtual addresses while the HCA has the ability to translate these addresses into physical
addresses in order to affect the actual message transfer.
Range Extenders
InfiniBand range extension is accomplished by encapsulating the InfiniBand traffic onto the
WAN link and extending sufficient buffer credits to ensure full bandwidth across the WAN.
Subnet Manager
The InfiniBand subnet manager assigns Local Identifiers (LIDs) to each port connected to the
InfiniBand fabric and develops a routing table based on the assigned LIDs. The IB Subnet Man-
ager is a concept of Software Defined Networking (SDN) which eliminates the interconnect
complexity and enables the creation of very large scale compute and storage infrastructures.
Switches
IB switches are conceptually similar to standard networking switches but are designed to meet IB
performance requirements. They implement flow control of the IB Link Layer to prevent packet
dropping, and to support congestion avoidance and adaptive routing capabilities, and advanced
Quality of Service. Many switches include a Subnet Manager. At least one Subnet Manager is
required to configure an IB fabric.
1.6 Support for Existing Applications and ULPs
IP applications are enabled to run over an InfiniBand fabric using IP over IB (IPoIB) or Ethernet
over IB (EoIB) or RDS ULPs. Storage applications are supported via iSER, SRP, RDS, NFS,