优化LSMT键值存储：提升写入性能

109 浏览量更新于2024-08-28 收藏 403KB PDF 举报

"提高基于LSMT的键值存储的写入性能" 在当今的高数据量和高并发的环境中，键值存储系统（Key-Value Store）因其高效的读写性能而被广泛应用于各种应用场景，尤其是那些对实时性要求高的系统。LSMT（Log-Structured Merge Tree）是一种针对键值存储设计的数据结构，它通过消除随机写入，优化磁盘I/O，从而提供出色的读取性能。然而，LSMT在长时间运行后，磁盘中的数据会需要定期进行合并（compaction）操作，这一过程会消耗大量的I/O资源，导致系统的整体性能受到影响。本文的研究者来自中国科学技术大学计算机科学与技术学院、安徽省高性能计算重点实验室以及国防科技大学高性能计算协同创新中心。他们针对LSMT的这一问题，提出了一个名为“Grouped Level Structure”的新设计。这个设计的目标是优化LSMT的compaction过程，以减少其对系统性能的影响。 Grouped Level Structure的核心思想是将数据分组并逐层合并，而不是一次性处理所有数据。这样的策略能够更好地管理I/O操作，减少单次compaction过程中涉及的数据量，从而降低对系统资源的占用。具体来说，这种方法可能包括以下几个关键点： 1. 数据分组：将待合并的数据划分为多个小组，每个小组内的数据可能在同一层或者相邻层，这样可以减小每次操作的复杂性。 2. 层级管理：通过优化不同层级之间的数据转移，使得compaction过程更加有序，避免大规模的数据迁移。 3. 并行处理：利用多线程或分布式计算资源，对不同小组的数据进行并行处理，提高compaction效率。 4. I/O调度：智能调度I/O操作，确保在不影响读取性能的同时，最大限度地减少写入时的I/O冲突。通过这些改进，Grouped Level Structure有望显著提高LSMT型键值存储的写入性能，减少对系统资源的占用，并且可能对整体系统的响应时间和稳定性带来积极影响。这项研究对于理解和改进基于LSMT的键值存储系统具有重要的理论和实践价值，为大数据时代的高性能存储解决方案提供了新的思路。

Improving Write Performance of LSMT-based Key-Value Store

WeiTao Zhang

1,2

, Yinlong Xu

1,3

, Yongkun Li

1,3

, Dinglong Li

1. School of Computer Science and Technology, University of Science and Technology of China

2. AnHui Province Key Laboratory of High Performance Computing, Hefei, China

3. Collaborative Innovation Center of High Performance Computing, National University of Defense Technology

Email: {avenger, ldlong}@mail.ustc.edu.cn, {ylxu, ykli}@ustc.edu.cn

Abstract—Key-value stores are widely used to provide much

higher read and write throughput than traditional SQL

databases. LSMT (log structure merge tree) based key-value

store, as one type of key-value stores, is applied in many

practical systems since it could eliminate random writes and

provide good read performance at the same time. However, the

data residing in disk needs compaction operations from time to

time, which takes a large amount of I/O resources. Since disk

access speed is much slower than DRAM and most data resides

in disks, the compaction operation will signiﬁcantly inﬂuence

the system performance. In this paper, we propose a grouped

level structure, which divides each level in LSMT into multiple

groups. Also, we propose a new compaction method for the

grouped level structure to reduce the compaction I/O overhead.

Our experiments show that the grouped level structure saves

about 55% to 78% I/O resource of compaction, so it improves

the write throughput by 69% to 284%, but only reduces

the read throughput by 5% to 9%. It improves the overall

throughput by 30% to 69% with read dominated workloads

of 25% write operations and 75% read operations.

Keywords-Key-Value, Log Structured Merge Tree, Com-

paction, Big Data

I. INTRODUCTION

With key-value store, data is represented as a collection

of key-value pairs, where the key of each value is unique. It

is a basic type of NoSQL database which doesn’t rely on the

traditional structures of relational database. Compared with

traditional relational database, key-value store is designed

to handle a huge amount of data and it has worked well

for shopping cart contents, landing page URLs, and default

account numbers, etc. Key-value stores are fast, scalable,

portable and ﬂexible, so they have been widely used in prac-

tical systems, including Google’s BigTable [6], Facebook’s

Cassandra [10], Apache Hbase [1], Amazon’s Dynamo [8]

and so on.

In some practical applications, key-value store faces write-

intensive workloads where the workloads are dominated by

data writes rather than reads [17], e.g. to store frequently-

changing objects [4, 5, 12]. To improve write performance,

many key-value stores are based on the log-structured merge

tree (LSMT) [13], like Hbase, Cassandra, BigTable and

LevelDB [2], which could be treated as a lightweight im-

plementation of BigTable and has been widely deployed in

many applications. Those key-value stores offer a similar

API to write data to and read data from the databases, and

the basic interfaces are put for write and get for read. To

support write intensive workloads, key-value stores usually

use append-only strategy to speed up write process, where

update and delete operations are performed as append writes.

To distinguish insert, delete and update operations, each key-

value pair is along with a ﬂag to identify those operations,

and the obsolete key-value pairs will be reclaimed by the

compaction operation.

The new coming key-value pairs in a LSMT key-value

stores are stored in a memory buffer at the beginning. When

the buffer is full, the key-value pairs are packed into an sst

ﬁle with sorted keys and persisted into an external storage.

Each sst ﬁle keeps a record of its minimum key and its

maximum key, which we call its key interval. To further

enhance read performance with spatial locality and temporal

locality of workloads, LSMT key-value stores divides all

key-value pairs into some levels. Usually there are multiple

times of sst ﬁles in level i +1 of which in level i. When

level i is full, all of its sst ﬁles are compacted into level

i +1.

With compaction, the keys in different sst ﬁles are sorted.

So apart from level 0, the key intervals of different sst ﬁles

in the same level do not overlap with each other. A given

key may falls into only one sst ﬁle in a level (apart from

level 0). With compaction, we can limit the sst ﬁles to be

searched for ﬁnding a given key. For example, in Figure 1,

key 7 falls in the key intervals of both ﬁles A and B,but

only B contains key 8. Suppose that we compact A

3,11

and

7,44

into ﬁles C

3,8

and D

11,44

. Then key 7 falls only in

the key interval of C, so we only need to search key 7 in

ﬁle C.

To read the value with a given key k, we will ﬁrstly

compare k with the minimum key and the maximum key

of an sst ﬁle. If k does not falls in its key interval, the

corresponding value is not in this ﬁle; otherwise, we will

use binary search to compare k with the keys in this ﬁle.

So with levelling and compaction, read latency in key-value

stores is limited.

In a long run, read latency in a key-store can be limited

by compaction. However, the compaction process itself will

consume a plenty of CPU and I/O resources, especially

for a write-intensive workload. In a LSMT key-value store,

most data resides in disk and access to disk is much slower

2016 IEEE 22nd International Conference on Parallel and Distributed Systems

DOI 10.1109/ICPADS.2016.77

553

下载后可阅读完整内容，剩余7页未读，立即下载

weixin_38677306

粉丝: 4
资源: 916

优化LSMT键值存储：提升写入性能

cpp-LevelDBGoogle开发的一个快速键值存储库

leveldb实现解析

lsmt：LSM树

小米Pegasus：高可用分布式KV存储系统设计揭秘

在设计一个高可用、高性能且具备强一致性的分布式键值存储系统时，应该如何考虑系统架构、数据分布、存储介质和一致性协议等因素？

IncompatibleClassChangeError(解决方案).md

中国智慧工地行业市场研究（2023）Word(63页).docx

java大题啊实打实的

asdjhfjsnlkdmv

二手车价格预测，代码核心任务是通过机器学习模型（如线性回归、随机森林和KNN回归）预测车辆的价格（current price），并使用评估指标（如 R² 和 MSE）来衡量不同模型的预测效果

最新资源