Accelerating Two-Dimensional Page Walks
for Virtualized Systems
Ravi Bhargava
Computing Solutions Group
Advanced Micro Devices
Austin, TX
ravi.bhargava@amd.com
Benjamin Serebrin
Computing Solutions Group
Advanced Micro Devices
Sunnyvale, CA
b enjamin.serebrin@amd.com
Francesco Spadini
Computing Solutions Group
Advanced Micro Devices
Austin, TX
francesco.spadini@amd.com
Srilatha Manne
Advanced Architecture & Technology Lab
Advanced Micro Devices
Bellevue, WA
srilatha.manne@amd.com
Abstract
Nested paging is a hardware solution for alleviating the software
memory management overhead imposed by system virtualization.
Nested paging complements existing page walk hardware to form
a two-dimensional (2D) pag e walk, which reduces the need for
hypervisor intervention in guest page table management. However,
the extra dimension also increases the maximum number of archi-
tecturally-required page table references.
This paper presents an in-depth examination of the 2D page
table walk overhead and options for decreasing it. These options
include using the AMD Opteron
TM
processor’s page walk cache
to exploit the strong reuse of page entry references. For a mix of
server and SPEC
R
benchmarks, the presented results show a 15%-
38% improvement in guest performance by extending the existing
page walk cache to also store the nested dimension of the 2D page
walk. Caching nested page table translations and skipping multiple
page entry references produce an additional 3%-7% improvement.
Much of the remaining 2D page walk overhead is due to low-
locality nested page entry references, which result in additional
memory hierarchy misses. By using large pages, the hypervisor can
eliminate many of these long-latency accesses and further improve
the guest performance by 3%-22%.
Categories and Subject Descriptors C.0 [General]: Modeling
of computer architecture; C.4 [Performance of Systems]: Design
studies; D.4.2 [Operating Systems]: Virtual Memory
General Terms Performance, Design, Measurement, Experimen-
tation
Keyw ords Virtualization, TLB, Memory Management, Nested
Paging, Page Walk Caching, Hypervisor, Virtual Machine Monitor,
AMD
1. Introduction
Virtualization allows multiple operating systems to run simulta-
neously on one physical system. These operating systems run as
guests on the virtualized system and have little or no knowledge
that they no longer control the physical system resources. The hy-
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. To copy otherwise, to republish, to post on servers or to redistribute
to lists, requires prior specific permission and/or a fee.
ASPLOS’08
March 1–5, 2008, Seattle, Washington, USA
Copyright
c
2008ACM978-1-59593-958-6/08/03...$5.00
pervisor is the underlying software that inserts abstractions into a
virtualized system: an operating system (OS) becomes a guest OS,
physical addresses become guest physical addresses, and, in gen-
eral, system elements that the OS presumed were real or physical
are converted into virtualized resources under control or manipula-
tion of the hypervisor [14, 16].
Ideally, a virtualized guest system will have comparable per-
formance to an equivalent native, non-virtualized system. This can
indeed be the case for compute-intensive applications. For exam-
ple, the performance overhead for a virtualized system running
SPECint
R
2000 benchmarks can be less than 5% because the hy-
pervisor is infrequently invoked [2]. However, as the number of op-
erations requiring hypervisor intervention increases, performance
can degrade substantially. While tolerable in many server consoli-
dation environments, these longer run times are unsatisfactory for
performance-sensitive applications.
Operations intercepted by the hypervisor in a virtualized system
could consume thousands of cycles of overhead to trap the condi-
tion, exit the guest, emulate the operation in the hypervisor, and re-
turn to the guest. These costs lead Adams and Agesen to state that
“reducing the frequency of exits is the most important optimization
for classical [hypervisors]” [2]. More specifically, one of the pri-
mary sources of virtualization exits is software memory translation
management, which is required to maintain the guest page tables.
AMD has implemented nested paging to greatly reduce the
overhead of hypervisor intervention in memory management [4].
Under nested paging, the guest controls its unmodified page tables.
However, what the guest considers to be real, or system, physical
addresses are in fact virtualized by the hypervisor. Each guest
physical address in the guest page table is looked up in the nested
pag e tables by hardware to obtain the system physical address. The
end result is a two-dimensional (2D) page walk that translates the
guest virtual address directly to the system physical address.
Although nested paging removes the overhead of hypervisor in-
tervention, it increases the maximum number of page entry ref-
erences architecturally required to generate a system physical ad-
dress. If a guest page walk has n levels and a nested page walk has
m levels, a 2D walk requires nm + n + m page entry references.
For example, a 2D page walk with four-level guest paging and four-
level nested paging has six times more page entry references than a
four-level native page walk. Therefore, the overall performance of
a virtualized system is improved by nested paging when the elimi-
nated hypervisor memory management overhead is greater than the
new 2D page walk overhead.
Translation look-aside buffers (TLBs) can limit the nested pag-
ing overhead by caching the full 2D translation and reducing the
frequency of page walks. For applications with a high TLB hit ratio,
the additional 2D latency will have a negligible impact. However,
26