capabilities and is not designed for efficient parallel dynamics simulation, which makes it difficult
to develop learning-based systems.
AirSim and CARLA: Both AirSim
1
[7] and CARLA [15] are open-source photo-realistic simula-
tors for autonomous vehicles built on Unreal Engine. CARLA is mainly made for autonomous driv-
ing research and only provides the dynamics of ground vehicles. Conversely, AirSim offers an inter-
face to configure multiple vehicle models for quadrotors and supports hardware-in-the-loop (HITL)
as well as software-in-the-loop (SITL) with flight controllers such as PX4. The vehicle is defined as
a rigid body whose dynamics model is simulated using NVIDIA’s physics engine PhysX, a popular
physics engine used by the large majority of today’s video games. However, this physics engine
is not specialized for quadrotors (or robots), and it is tightly coupled with the rendering engine to
allow simulating environment dynamics. Because of this rigid connection between rendering and
physics simulation, AirSim can achieve only limited simulation speeds. This limitation makes it
difficult to apply the simulator to challenging model-free reinforcement learning tasks, e.g. training
an end-to-end control policy for quadrotor stabilization under harsh initialized poses [16] or flying
through a fast moving gate [17].
FlightGoggles: FlightGoggles [4] is a photo-realistic sensor simulator for perception-driven robotic
vehicles. FlightGoggles consists of two separate components: a photo-realistic rendering engine
built on Unity and a quadrotor dynamics simulation implemented in C++. In addition, it also pro-
vides an interface with real-world vehicles and actors in a motion capture system. FlightGoggles is
very useful for rendering camera images given trajectories and inertial measurements from flying ve-
hicles in real-world, in which the collected dataset [18] is used for testing vision-based algorithms.
Flightmare shares the same motivation with FlightGoggles of decoupling the dynamics modeling
from the photo-realistic rendering engine. However, our simulator offers a larger suite of sensors
observations, an API to extract the environment point-cloud, and a more structured physics inter-
face, which allows simulating multiple agents in parallel. These characteristics open up several new
opportunities for both robotics and machine learning research (see Section 4).
Apart from the aforementioned simulators, there are many more existing simulators that have been
widely adopted by other research communities. For example, MuJoCo [1] has been widely used
by the reinforcement learning community for benchmark comparisons. Similarly, RaiSim [2] is a
physics engine for robotics and AI research written in C++, that supports massive parallel dynamics
simulation. However, both simulators do not support complex 3D environments and photo-realistic
image rendering. Sim4CV [19] is a photo-realistic simulator but made solely for computer vision
applications.
3 Methodology
3.1 System Overview
Flightmare is a modular and flexible quadrotor simulator that is mainly composed of two separate
components: a photo-realistic rendering engine built with the Unity Editor and a quadrotor dy-
namics simulation. We decouple the quadrotor’s dynamic modeling from the rendering engine in
order to achieve fast and accurate dynamics simulation by making use of parallel programming [20].
Flightmare provides a flexible interface for the user to simulate different sensors in various complex
close-to-reality 3D environments. The interface between the rendering engine and the quadrotor
dynamics is implemented using a high-performance asynchronous messaging library ZeroMQ
2
. In
addition, we use the python wrapper [21] to implement OpenAI-Gym style interface for reinforce-
ment learning tasks. A system overview of Flightmare is shown in Figure 1.
3.2 Rendering Engine
3D Environments. The rendering engine of Flightmare was built with Unity, which is a popular
cross-platform game engine and a general platform for artificial intelligence [5]. Unity enables high-
fidelity graphical rendering, including realistic pre-baked or real-time lighting, flexible combinations
of different meshes, materials, shaders, and textures for 3D objects, skyboxes for generating realistic
1
AirSim also has an experimental Unity release.
2
https://zeromq.org
3