Xen
Dom0
Kernel
VM
Userspace Userspace
Kernel
EL0
EL1
EL2
Backend
Device
Frontend
Driver
Virtio I/O
vGIC
Figure 2: Xen ARM Architecture
Host OS
Kernel
KVM
KVM
VM
Userspace Userspace
Kernel
EL0
EL1
EL2
vGIC
Virtio
Device
Virtio
Driver
Virtio I/O
Figure 3: KVM ARM Architecture
come the limitation that the ARM architecture was not
classically virtualizable [8]. All server and networking class
ARM hardware is expected to implement these extensions.
We provide a brief overview of the ARM hardware vir-
tualization extensions and how hypervisors leverage these
extensions, focusing on ARM CPU virtualization support
and contrasting it to how x86 works.
The ARM virtualization extensions are centered around
a new CPU privilege level (also known as exception level),
EL2, added to the existing user and kernel levels, EL0 and
EL1, respectively. Software running in EL2 can configure
the hardware to support VMs. To allow VMs to interact
with an interface identical to the physical machine while
isolating them from the rest of the system and preventing
them from gaining full access to the hardware, a hypervisor
enables the virtualization features in EL2 before switching to
a VM. The VM will then execute normally in EL0 and EL1
until some condition is reached that requires intervention of
the hypervisor. At this point, the hardware traps into EL2
giving control to the hypervisor, which can then interact
directly with the hardware and eventually return to the VM
again. When all virtualization features are disabled in EL2,
software running in EL1 and EL0 works just like on a system
without the virtualization extensions where software running
in EL1 has full access to the hardware.
ARM hardware virtualization support enables traps to EL2
on certain operations, enables virtualized physical memory
support, and provides virtual interrupt and timer support.
ARM provides CPU virtualization by allowing software in
EL2 to configure the CPU to trap to EL2 on sensitive
instructions that cannot be safely executed by a VM. ARM
provides memory virtualization by allowing software in
EL2 to point to a set of page tables, Stage-2 page tables,
used to translate the VM’s view of physical addresses to
machine addresses. When Stage-2 translation is enabled,
the ARM architecture defines three address spaces: Virtual
Addresses (VA), Intermediate Physical Addresses (IPA), and
Physical Addresses (PA). Stage-2 translation, configured in
EL2, translates from IPAs to PAs. ARM provides interrupt
virtualization through a set of virtualization extensions to
the ARM Generic Interrupt Controller (GIC) architecture,
which allows a hypervisor to program the GIC to inject
virtual interrupts to VMs, which VMs can acknowledge
and complete without trapping to the hypervisor. However,
enabling and disabling virtual interrupts must be done in
EL2. Furthermore, all physical interrupts are taken to EL2
when running in a VM, and therefore must be handled
by the hypervisor. Finally, ARM provides a virtual timer,
which can be configured by the VM without trapping to the
hypervisor. However, when the virtual timer fires, it raises a
physical interrupt, which must be handled by the hypervisor
and translated into a virtual interrupt.
ARM hardware virtualization support has some similari-
ties to x86
1
, including providing a means to trap on sensitive
instructions and a nested set of page tables to virtualize
physical memory. However, there are key differences in how
they support Type 1 and Type 2 hypervisors. While ARM
virtualization extensions are centered around a separate CPU
mode, x86 support provides a mode switch, root vs. non-root
mode, completely orthogonal from the CPU privilege rings.
While ARM’s EL2 is a strictly different CPU mode with its
own set of features, x86 root mode supports the same full
range of user and kernel mode functionality as its non-root
mode. Both ARM and x86 trap into their respective EL2
and root modes, but transitions between root and non-root
mode on x86 are implemented with a VM Control Structure
(VMCS) residing in normal memory, to and from which
hardware state is automatically saved and restored when
switching to and from root mode, for example when the
hardware traps from a VM to the hypervisor. ARM, being
a RISC-style architecture, instead has a simpler hardware
mechanism to transition between EL1 and EL2 but leaves it
up to software to decide which state needs to be saved and
restored. This provides more flexibility in the amount of
work that needs to be done when transitioning between EL1
and EL2 compared to switching between root and non-root
mode on x86, but poses different requirements on hypervisor
software implementation.
ARM Hypervisor Implementations. As shown in Fig-
ures 2 and 3, Xen and KVM take different approaches to
using ARM hardware virtualization support. Xen as a Type 1
hypervisor design maps easily to the ARM architecture, run-
1
Since Intel’s and AMD’s hardware virtualization support are very
similar, we limit our comparison to ARM and Intel.