数据库管理系统的硬件挑战与优化

需积分: 9 98 浏览量更新于2024-07-19 收藏 3.67MB PDF 举报

"MC.Databases.on.Modern.Hardware英文版" 现代数据库系统在处理日益增长的数据量和复杂查询时面临着巨大的挑战，这需要数据库管理系统(DBMS)具备更强的处理能力。随着现代硬件提供了更高的并行性和功能，如何充分利用这些资源一直是几十年来的持续难题。特别是，硬件的发展对系统的可扩展性提出了三个主要挑战： 1. **利用多核提供的大量线程级并行性**：现代处理器通常包含多个核心，能够同时执行多个线程。数据库管理系统需要有效地分配任务到这些核心，以实现高性能并行处理。 2. **在多插槽多核系统中实现可预测的高效执行**：由于不同插槽和核心之间的性能差异，系统可能面临非均匀性问题。软件需要设计得足够智能，以适应这种不一致性，确保性能稳定。 3. **利用激进的微架构特性**：现代处理器包含多种优化技术，如超线程、高速缓存层次结构和预取机制。数据库系统需要理解和利用这些特性，以提高数据访问速度和内存管理效率。这本书深入探讨了这三个挑战，并综述了最近提出的解决方案，旨在缓解这些问题。作者Anastasia Ailamaki、Erietta Liarou、Pinar Tozun、Danica Porobic和Iraklis Psaroudakis通过《Databases on Modern Hardware: How to Stop Underutilization and Love Multicores》阐述了如何优化数据库系统，充分利用多核硬件的潜力，避免资源的低效使用。书中的内容可能涵盖了写前日志（Write-Ahead Logging）在即时恢复、系统重启、媒体恢复和系统故障切换中的应用，以及数据管理的其他关键领域，包括查询语言、数据库系统架构、事务管理、数据仓库、XML与数据库集成、数据流系统、大规模数据分布、多媒体数据管理以及数据挖掘等。通过这本书，读者可以了解到如何在面对现代硬件的复杂性时，设计和优化数据库管理系统，以达到最佳性能和可扩展性，从而克服上述挑战。这是一本针对数据管理和数据库系统研究人员、开发者以及对数据库性能优化感兴趣的读者的重要参考资料。

2 1. INTRODUCTION

Figure 1.1: e implicit/vertical scalability dimension refers to optimizing the performance of

DBMS by increasing the utilization of cores and caches.

Figure 1.2: e explicit/horizontal scalability dimension refers to optimizing the performance of

DBMS by better utilizing the increasing number of cores and sockets.

half of the execution time goes to memory stalls when running data-intensive workloads [42].

As a result, on processors that have the ability to execute four instructions per cycle (IPC),

which is common for modern commodity hardware, data intensive workloads, especially trans-

action processing, achieve around one instruction per cycle [141, 152]. Such underutilization of

microarchitectural features is a great waste of hardware resources.

Several proposals have been made to reduce memory stalls through improving instruc-

tion and data locality to increase cache hit rates. For data, these range from cache-conscious

data structures and algorithms [30] to sophisticated data partitioning and thread schedul-

ing [115]. For instructions, they range from compilation optimizations [131], and advanced

prefetching [43], to computation spreading [13, 26, 150] and transaction batching for instruc-

1.2. EXPLICIT/HORIZONTAL DIMENSION 3

tions [14, 55]. In addition, several recent proposals opt for hardware specialization for some of

the database operations [65, 85, 166].

1.2 EXPLICIT/HORIZONTAL DIMENSION

Figure 1.2 illustrates the explicit/horizontal dimension of multicore hardware. e scalability of

this dimension refers to utilizing the increasing number of cores and sockets in a single multi-

socket multicore server hardware.

Since the beginning of this decade, power draw and heat dissipation prevent processor

vendors from relying on rising clock frequencies or more aggressive microarchitectural tech-

niques for higher performance. Instead, they add more processing cores or hardware contexts

on a single processor to enable exponentially increasing opportunities for parallelism [107]. Ex-

ploiting this parallelism is crucial for utilizing the available architectural resources and enabling

faster software. However, designing scalable systems that can take advantage of the underly-

ing parallelism remains a challenge. In traditional high-performance transaction processing, the

inherent communication leads to scalability bottlenecks on today’s multicore and multisocket

hardware. Even systems that scale very well on one generation of multicores might fail to scale-

up on the next generation. On the other hand, in traditional online analytical processing, the

database operators that were designed for unicore processors fail to exploit the abundant paral-

lelism oﬀered by modern hardware.

Servers with multiple processors and non-uniform memory access (NUMA) design

present additional challenges for data management systems, many of which were designed with

implicit assumptions that core-to-core communication latencies and core-to-memory access la-

tencies are constant regardless of location. However, today for the ﬁrst time we have Islands,

i.e., groups of cores that communicate fast among themselves and slower with other groups.

Currently, an Island is represented by a processor socket but soon, with dozens of cores on the

same socket, we expect that Islands will form within a chip. Additionally, memory is accessed

through memory controllers of individual processors. In this setting, memory access times vary

greatly depending on several factors including latency to access remote memory and contention

for the memory hierarchy such as the shared last level caches, the memory controllers, and the

interconnect bandwidth.

Abundant parallelism and non-uniformity in communication present diﬀerent challenges

to transaction and analytical workloads. e main challenge for transaction processing is com-

munication. In this part of the book, we initially teach a methodology for scaling up transaction

processing systems on multicore hardware. More speciﬁcally, we identify three types of com-

munication in a typical transaction processing system: unbounded, ﬁxed, and cooperative [67]. We

demonstrate that the key to achieving scalability on modern hardware, especially for transaction

processing systems, but also for any system that has similar communication patterns, depends

on avoiding the unbounded communication points or downgrading them into ﬁxed or coopera-

4 1. INTRODUCTION

tive ones. We show how eﬀective this methodology is in practice by surveying related proposals

from recent work (e.g., [36, 76, 109, 146, 149, 158]).

Non-uniform communication latencies make it appealing to regard multisocket as a dis-

tributed system and deploy multiple nodes in a shared-nothing conﬁguration [76, 146]. While

this approach works great for perfectly partitionable workloads, it is very sensitive to distributed

transactions and the workload skew. At the same time, hardware-oblivious shared-everything

systems suﬀer from non-uniform latencies that amplify bottlenecks in the critical path [116]. In

order to achieve scalability on multisockets one needs to make the system aware of the hardware

topology and dynamically adapt to workload and hardware [115].

On the other hand, traditional online analytical processing workloads are formed of scan-

heavy, complex, ad-hoc queries that do not suﬀer from the unbounded communication as in

transaction processing. Analytical workloads are still concerned with the variability of latency,

but also with avoiding saturating resources such as memory bandwidth. In many analytical work-

loads that exhibit similarity across the query mix, sharing techniques can be employed to avoid

redundant work and re-use data in order to better utilize resources and decrease contention. We

survey recent techniques that aim at exploiting work and data sharing opportunities among the

concurrent queries (e.g., [22, 48, 56, 119]).

Furthermore, another important aspect of analytical workloads, in comparison to trans-

action processing, is the opportunity for intra-query parallelism. Typical database operators,

such as joins, scans, etc., are mainly optimized for single-threaded execution. erefore, they

fail to exploit intra-query parallelism and cannot utilize several cores naïvely. We survey recent

parallelized analytical algorithms on modern non-uniform, multisocket multicore architectures

[9, 15, 94, 127].

Finally, in order to optimize performance on non-uniform platforms, the execution engine

needs to tackle two main challenges for a mix of multiple queries: (a) employing a scheduling

strategy for assigning multiple concurrent threads to cores in order to minimize remote mem-

ory accesses while avoiding contention on the memory hierarchy; and (b) dynamically deciding

on the data placement in order to minimize the total memory access time of the workload.

e two problems are not orthogonal, as data placement can aﬀect scheduling decisions, while

scheduling strategies need to take into account data placement. We review the requirements and

recent techniques for highly concurrent NUMA-aware scheduling for analytics, which take into

consideration data locality, parallelism, and resource allocation (e.g., [34, 35, 91, 122]).

1.3 STRUCTURE OF THE BOOK

In this book, we aim to examine the following questions.

• How can one adapt traditional execution models to fully exploit modern hardware?

• How can one maximize data and instruction locality at the right level of the memory

hierarchy?

剩余114页未读，继续阅读

wzzf819

粉丝: 3
资源: 28

数据库管理系统的硬件挑战与优化

Wrox.Beginning.Visual.Basic.2005.Databases.Nov.2005.rar

怎么查询 sys.databases 目录视图，语句和操作是什么

在sqlserver中如何查找实际的数据库名称？

SSMS中EXEC中对登录用户的操作语句

Unable to attach database file /data/data/com.example.bigwork/databases/scoring.db: [SQLITE_CANTOPEN] Unable to open the database file (unable to open database: /data/data/com.example.bigwork/databases/scoring.db)

sqlserver查询所有库名

我在sys.dm_os_performance_counters里没有找到database_id这个对象

Python 模型定义 ：通过pymysql和数据模型(models.py)创建mysql表及表结构

最新资源

Python 模型定义：通过pymysql和数据模型(models.py)创建mysql表及表结构