Figure 2: VM exits in stock Xen 4.0 during server bootup and runtime of a guest VM.
host OS may also include drivers for the physical devices
which are used by the system (
4
). An emulator which em-
ulates system devices that guest VMs interact with is also
commonly run within the host OS. All together, this func-
tionality req uires over a million lines of code.
We now dicuss the roles of the hypervisor in emulating and
managing th e main resources in today’s computer systems.
Processor cores: The first responsibility of any virtualiza-
tion solution is the ability to arbitrate access to the processor
cores as well as sometimes emulate their functionality. The
hypervisor needs to cope with the dyn amic addition and re-
moval of VMs as they get started and stopped, as well as
the ability to schedule multiple VMs on the same system. In
addition to scheduling the VMs, the virtualization layer also
uses the scheduler to periodically, every clock tick, run some
of its own functionality – for example the hypervisor need s
to check whether to deliver timer interrupts to the VM in
this clock tick. The virtualization layer may also emulate
the underlying processor as the presented processor cores
may differ in capabilities from the actual cores.
Memory: The hypervisor is also in charge of managing the
host physical memory of the system. Each VM is assigned
some guest physical memory that the guest OS can man-
age and assign as virtual memory to the applications. Here,
the virtualization layer can take a number of different ap-
proaches to managing the allocation of host physical memory
among the various VMs. One popular use of virtualization is
server consolidation which relies on the virtu alization layer
to dynamically manage the allocation of physical memory
to the different VMs in the system.
I/O devices: Another important functionality of the virtu-
alization layer is dealing with I/O devices such as a network
interface card (N IC). Today’s virtualization solutions com-
monly create virtual devices that they assign to the guest
VMs. The v irtualization layer needs to manage these virtual
devices and emulate their behavior. There is also the option
of dedicating physical devices to the VMs. In t his situation,
the virtualization layer only needs to assign and reclaim the
devices. It does not need to emulate their behavior. In both
cases, however, the emulator is needed to emulate the PCI
(Peripheral Component Interconnect) bus. The guest OS
interacts with the PCI bus d uring bootup to discover and
configure the available devices.
Interrupts and Timers: The virtualization layer must also
emulate the interrupt subsystem and timers for the guest
VMs.
For interrupts, the main parts of the underlying hard-
w
are interrupt subsystem emulated by the hypervisor in-
clude the I/O APIC
1
(which routes interrupts from devices
to processor cores) and local APICs (which are attached
to each core as an interface to the interrupt subsystem for
sending and receiving interrupt messages to and form other
local APICs or the I/O APIC). These APICs are emulated
so the guest OS can use them to configure and manage in-
terrupts inside the VM, while physical APICs are controlled
by the hypervisor for the management of the physical plat-
form.
For timers, since the hypervisor utilizes the physical
t
imer devices to get periodic clock ticks itself, it has to em-
ulate the timer device for each guest VM. Today, VMs can
be scheduled on different cores and as such, the interrupts
and timers must first go through the hypervisor which will
deliver th e interrupts to the VM by utilizing its knowledge
of the VM’s current location.
2.2 Attack Surface
The v irtualization layer is heavily involved throughout the
lifetime of the guest VM. Each interaction between the VM
and the hypervisor is then a potential attack vector that
could be exploited by the guest. The guest O S interacts di-
rectly with the hypervisor (
5
) and indirectly with the host
OS and the emulator (
6
) through VM exits (and hyper-
calls if paravirtualization is used). A VM exit is an event
which occurs when the VM’s code is interrupted and the hy-
pervisor code begins to execute to h andle some event (e.g.,
emulate memory access, deliver a virtual timer interrupt,
etc.). A hypercall is similar to a system call and is used by
guest VMs to request explicit service from the hypervisor.
Hypercalls are not considered further as we are not using
paravirtualization. VM exits are quite frequent even when
the guest OS inside the VM is not doing any work; in an idle
VM running on top of Xen 4.0, the VM exits occur ∼600
times per second.
On a 64-bit Intel x86 architecture with virtualization ex-
tensions, there are 56 reasons for VM exits, and this forms
the large attack surface which is the basis of the security
threat.
The reasons VM for exits are described in Table
1
. Each VM exit causes the hypervisor code to run so the
hypervisor can intervene when the guest OS performs some
operation that caused the associated exit. This allows the
hypervisor to maintain the abstraction of the system which
it presents to the guest OSes, for example it can return dif-
ferent CPUID values than the values actually reported by
the hardware.
In order to emphasize how often the large and complex
1
A
PIC is the advanced programmable interrupt controller.