RDMA技术：革新云计算的数据管理

需积分: 29 85 浏览量更新于2024-07-18 1 收藏 4.27MB PDF 举报

本文档是一篇关于RDMA（远程直接内存访问）技术的专题文章，收录在2017年3月的《数据工程技术委员会公告》第40卷第1期的IEEE计算机学会通信中。该期特别关注分布式数据管理中RDMA的应用及其所带来的变革。首先，编辑部的信件由David Lomet撰写，概述了本期特刊的主题背景和重要性，提及了RDMA作为新一代网络通信协议，正在重塑数据中心的架构和性能优化。Aleksandar Dragojević、Dushyanth Narayanan和Miguel Castro撰写的“RDMA读：使用还是不使用？”探讨了在高吞吐量网络环境中，是否应该广泛采用RDMA技术，以及它如何影响数据传输效率和系统资源利用。设计未来高性能网络数据库的文章由Claude Barthels、Gustavo Alonso和Torsten Hoefer合作完成，他们讨论了如何利用RDMA来优化数据库设计，提升数据处理速度和可靠性。Abdallah Salama等人提出的“重新思考高速网络上的分布式查询执行”研究了如何通过RDMA改进分布式查询处理的性能和响应时间。 Patrick Stuedi等人的“Crail：分布式数据处理的高性能I/O架构”介绍了基于RDMA的新型存储和I/O模型，旨在提高大规模数据处理的效率。Xiaoyi Lu和Dipti Shankar等人则展示了如何通过RDMA-Memcached实现可扩展的键值存储系统，支持分布式数据管理。 Chinmay Kulkarni等人在“Beyond Simple Request Processing with RAM Cloud”中探讨了如何利用RDMA超越传统请求处理方式，提升云服务的性能。最后，会议和期刊通知部分提到了ICDE 2017（国际数据工程大会）的相关信息，以及TCDE（技术委员会数据工程）的成员动态。这篇专题文章详细探讨了RDMA技术在云计算平台中的应用，包括其对数据传输、数据库设计、分布式查询执行、I/O架构以及分布式数据存储的影响。这些研究不仅展示了RDMA技术的潜力，也预示着其在未来的数据中心和云计算环境中的核心地位。

Reliable connections Unreliable datagrams

Scalability Limited Good

Congestion control Supported No

Reliable Yes No

In-order delivery Yes No

Message size 2 GB 2 kB

RDMA reads and writes Yes No

Table 1: RDMA transport capabilities

To solve this, we are planning to extend FaRM to invoke application-deﬁned callbacks with addresses of objects

that should be moved for de-fragmentation and have the application move the objects. To obtain the space for the

new object during de-fragmentation, the application can simply invoke FaRM’s allocator. The allocator always

prefers mostly full slabs for new allocations to reduce fragmentation.

Dealing with de-fragmentation at the application level in this way instead of performing it transparently by

the system can be beneﬁcial as application-level knowledge can be used to optimize the process. For instance, the

application can choose to postpone de-fragmentation to avoid increasing load in busy periods, or to completely

ignore the callback if it will soon need to allocate objects of the same size.

The application needs more information about object pointers to perform de-fragmentation than for resizing—

the callback is invoked just with the address and size of the object to be moved, without providing a path to the

object as is the case when resizing. This means that there needs to be a simple way to ﬁnd the pointers to the

object given just its address. For instance, to support de-fragmentation, the application might need to use a

doubly-linked list where it would otherwise use a singly-linked list.

Cold objects. Despite the decreasing price of main memory, it still remains much more expensive than SSDs

and disks. To reduce costs, some applications store only frequently accessed objects in memory and keep the

rest on disk. A system that uses RDMA reads needs to solve two challenges to support this kind of tiering:

identifying cold objects and evicting them to the storage tier. Identifying cold objects is hard as the CPU on the

host machine does not observe remote accesses and the NIC does not keep detailed access stats. Alternatively,

the application can decide which objects to move to the cold tier and when using application-level semantics. For

instance, the application might periodically evict objects older than some threshold. Objects can be moved in a

similar fashion to de-fragmentation—pointers to the object are either replaced by cold-tier pointers or invalidated

when the objects is evicted.

7 Future challenges

7.1 Networking

Table 1 summarizes the advantages and drawbacks of using RDMA RC and UD transports, as discussed in

Section 3. It shows that neither transport provides all desired features. UD provides good scalability, but it

leaves dealing with congestion control, re-transmissions, and packetization and re-assembly to software. It also

does not support RDMA reads and writes. On the other hand, RC has scalability issues because of limitations

of available hardware. Building a network transport that provides all desired features remains a challenge.

To build it, we need to ﬁrst rethink what the important features are. Not all features from Table 1 are equally

important and a simpler transport can be more efﬁcient. For instance, most distributed systems robustly deal with

reordered messages, so providing in-order delivery guarantees in the network is not useful. Re-transmission can

be more efﬁcient when message semantics are known, so it makes sense to either implement it in software or

expose the semantics to hardware. For instance, idempotent operations, like a read-only RPC, can be safely

executed multiple times, so there is no need to reliably respond to it, as the operation can simply be re-executed

if the response is lost. It might also be viable to build hardware support for the common case, and leave corner

cases to software. For instance, optimizing hardware for RDMA reads that ﬁt in a network packet or even not

supporting bigger RDMA reads at all would be acceptable to many applications.

NIC hardware can be improved to reduce the CPU cost of issuing RDMA operations regardless of the

abstraction it implements. The predominant overheads when interacting with the NIC are PCI transactions used

to issue and complete operations. These overheads can be reduced by batching—issuing or completing multiple

operations with a single PCI transaction [6]. There are opportunities to allow more batching than today, for

instance by allowing applications to issue batches containing operations on different connections, which is not

supported today. Alternatively, instead of waiting for PCI messages from the CPU, the NIC could periodically

poll request queues in memory. This would remove the need for the CPU to initiate PCI transactions and CPU

overheads of initiating them, but it would also increase latency, so it might not always be a good trade-off.

7.2 Hardware acceleration

Widespread hardware acceleration is going to change the way we think about and build systems in the future.

With the end of Moore’s law nearing, big companies are heavily investing in hardware acceleration. For instance,

Microsoft is deploying FPGA Catapult boards [18, 3], Google is deploying custom TPU ASICs to speed up

machine learning [31], Intel has announced work on integration of Xeon processors with coherent FPGAs in a

single package [26], and Mellanox has a product brief for a ConnectX-3 Pro card with FPGA [32]. This creates

an opportunity to build hardware that makes distributed systems better.

The main challenge with hardware specialization is in selecting the right primitives to move to hardware,

because of cost and complexity of development, even when using FPGAs. For this reason, primitives like

encryption and compression are the best targets—they are relatively simple and they would beneﬁt many appli-

cations. Time synchronization, which would make time-based protocols better, or remote read tracking, which

could help with detecting cold or hot objects, are also good candidates, as they would be useful in many cases.

Implementing parts of a general, widely-used distributed computation platform in hardware would also bene-

ﬁt many applications. We have experimented with hardware implementation of FaRM atomic reads and have

seen relatively small but consistent performance improvements. Other operations used by such a platform like

pointer chain following, hashtable lookups, and even parts of transaction commit protocol are good candidates

for acceleration in hardware.

With the right hardware primitives system builders would not have to choose between performance and

ﬂexibility, like they do today with RDMA reads. A system that stores data in a hashtable and implements

hashtable lookups in the NIC would give us the best of both worlds—performance of hardware lookups and

ﬂexibility of RPC-only systems. Some work on ofﬂoading hashtable functionality to FPGA has already been

done and the results are promising [8, 9]. The main remaining questions are what the right software-hardware

design is, how close to today’s NIC’s request rates can performance of a hardware hashtable get and how much

ﬂexibility will have to be traded off to support efﬁcient hardware implementation.

8 Conclusion

So, should you use RDMA reads or not? As always, the answer is: it depends on the workload. RDMA reads

can deliver beneﬁts both in terms of latency and throughput, but they are less ﬂexible than RPCs. We discussed

ways to overcome these ﬂexibility limitations in applications with well deﬁned schemas and data layout like

relational and graph databases. RDMA reads are a good choice for those applications. On the other hand, using

RDMA reads complicates the system and puts additional requirements on the application, at the extreme wasting

space and burning through some of the performance beneﬁt it brings. In the future we might have it all—a fast

hashtable implemented in hardware can give us performance of hardware and ﬂexibility of a key-value store, but

this will require overcoming a number of challenges.

References

[1] B. Atikoglu, Y. Xu, E. Frachtenberg, S. Jiang, M. Paleczny. Workload Analysis of a Large-scale Key-value Store.

SIGMETRICS 2012.

[2] V. Basant, P. Rahoul. Oracle’s Sonoma processor: Advanced low-cost ‘ processor for enterprise workloads.

HOTCHIPS 2015.

[3] A. Caulﬁeld, E. Chung, A. Putnam, H. Angepat, J. Fowers, M. Haselman, S. Heil, M. Humphrey, P. Kaur, J.-Y. Kim,

D. Lo, T. Massengill, K. Ovtcharov, M. Papamichael, L. Woods, S. Lanka, D. Chiou, D. Burger. A Cloud-Scale

Acceleration Architecture. MICRO 2016.

[4] A. Dragojevi

c, D. Narayanan, M. Castro, O Hodson. FaRM: Fast Remote Memory. NSDI 2014.

[5] A. Dragojevi

c, D. Narayanan, E. B. Nightingale, M. Renzelmann, A. Shamis, A. Badam, M. Castro. No Compro-

mises: Distributed Transactions with Consistency, Availability, and Performance. SOSP 2015.

[6] M. Flajslik, M. Rosenblum. Network Interface Design for Low Latency Request-Response Protocols. USENIX ATC

2013.

[7] S. Hudgens. Overview of Phase-Change Chalcogenide Nonvolatile Memory Technology. MSR bulletin, 2004, vol

29, number 11.

[8] Z. Istv

an, D. Sidler, G. Alonso. Building a distributed key-value store with FPGA-based microservers. FPL 2015.

[9] Z. Istv

an, D. Sidler, G. Alonso, M. Blott, K. Vissers. A Flexible Hash Table Design For 10Gbps Key-value Stores on

FPGAs. FPL 2013.

[10] J. Ousterhout, A. Gopalan, A. Gupta, A. Kejriwal, C. Lee, B. Montazeri, D. Ongaro, Diego and S. J. Park, H. Qin,

M. Rosenblum, S. Rumble, R. Stutsman, S. Yang. The RAMCloud Storage System. TOCS, September 2015, vol 33,

number 3.

[11] A. Kalia, M. Kaminsky, D. G. Andersen. FaSST: Fast, Scalable, and Simple Distributed Transactions with Two-Sided

(RDMA) Datagram RPCs. OSDI 2016.

[12] A. Kalia, M. Kaminsky, D. G. Andersen. Using RDMA Efﬁciently for Key-Value Services. SIGCOMM 2014

[13] C. Lee, S.J. Park, A. Kejriwal, S. Matsushita, J. Ousterhout. Implementing Linearizability at Large Scale and Low

Latency. SOSP 2015.

[14] C. Mitchell, G. Yifeng, L. Jinyang. Using one-sided RDMA reads to build a fast, CPU-efcient key-value store.

USENIX ATC 2013.

[15] R. Mittal, T. Lam, N. Dukkipati, E. Blem, H. Wassel, M. Ghobadi, A. Vahdat, Y. Wang, D. Wetherall, D. Zats.

TIMELY: RTT-based Congestion Control for the Datacenter. SIGCOMM 2015.

[16] S. Neuvonen, A. Wolski, M. Manner, V. Raatikka. Telecom Application Transaction Processing benchmark. http:

//tatpbenchmark.sourceforge.net.

[17] D. Ongaro, S. M. Rumble, R. Stutsman, J. Ousterhout, M. Rosenblum. Fast Crash Recovery in RAMCloud. SOSP

2011.

[18] A. Putnam, A. Caulﬁeld, E. Chung, D. Chiou, K. Constantinides, J. Demme, H. Esmaeilzadeh, J. Fowers, J. Gray,

M. Haselman, S. Hauck, S. Heil, A. Hormati, J.-Y. Kim, S. Lanka, E. Peterson, A. Smith, J. Thong, P. Y. Xiao, D.

Burger, J. Larus, G. P. Gopal, S. Pope. A Reconﬁgurable Fabric for Accelerating Large-Scale Datacenter Services.

ISCA 2014.

[19] S. M. Rumble, A. Kejriwal, J. Ousterhout. Log-structured Memory for DRAM-based Storage. FAST 2011.

剩余73页未读，继续阅读

sujingxd

粉丝: 1
资源: 2

RDMA技术：革新云计算的数据管理

c语言利用RDMA 实现客户端和服务器端发送接收消息

c语言利用RDMA技术实现客户端和服务器端互相读取对方内存

RDMA编程用户手册-官方中文版

Accelerating Redis with RDMA over InfiniBand

Bypassing RDMA Security Mechanisms.pdf

A study of iSCSI extensions for RDMA (iSER)

Storage protocol designs - A study of iSCSI extensions for RDMA

RDMA原理分析、对比和技术实现解析.zip

Octopus: an RDMA-enabled Distributed Persistent Memory File System

基于RDMA的分布式图探索的快速和并行RDF查询

最新资源