A Survey of Software Testing in the Cloud
Koray
˙
Inc¸ki
∗†
,
˙
Ismail Arı
†
, Hasan S
¨
ozer
†
∗
T
¨
UB
˙
ITAK B
˙
ILGEM, Information Technologies Institute, Kocaeli, Turkey
koray.incki@bte.tubitak.gov.tr
†
Computer Engineering Department,
¨
Ozye
˘
gin University,
˙
Istanbul, Turkey
{koray.incki, ismail.ari, hasan.sozer}@ozyegin.edu.tr
Abstract—Cloud computing has emerged as a new computing
paradigm that impacts several different research fields, including
software testing. Testing cloud applications has its own pecu-
liarities that demand for novel testing methods and tools. On
the other hand, cloud computing also facilitates and provides
opportunities for the development of more effective and scalable
software testing techniques. This paper reports on a systematic
survey of published results attained by the synergy of these
two research fields. We provide an overview regarding main
contributions, trends, gaps, opportunities, challenges and possible
research directions. We provide a review of software testing over
the cloud literature and categorize the body of work in the field.
Index Terms—cloud computing; software testing; cloud testing;
cloud-based software testing; testing cloud services; testing as a
service
I. INTRODUCTION
Cloud computing has emerged as a new computing
paradigm that facilitates the development and utilization of
highly flexible, elastic services on-demand, and over broad-
band network access. Those attributes are driving many orga-
nizations to move their businesses to a cloud platform.
Software testing has been one of the best practice areas
for migrating to cloud environment. Virtualization, which is
an enabling technology of cloud computing, was first used
for quickly creating virtual computing resources with different
operating systems (OS) to test software applications on various
platforms [1]. Testing new software often requires costly
server, storage and network devices only for a limited time [2].
These computing resources are either not used or underutilized
after testing, thus incurring extra cost on budget.
Especially in some application domains, software testing
requires extensive resources. For example, to test the perfor-
mance and scalability of a banking application, the system
must be stressed with requests from millions of users in a
short time interval. This is a realistic scenario that should be
tested because people rush to their bank accounts regularly on
every payday. Reproducing such a scenario would require the
provider to set up a test harness (including the user databases)
to emulate the actions of millions of users. Similarly, mobile
application providers frequently have to deal with maintain-
ing the quality of their services over a plethora of various
combinations of platforms [3]. The computing platforms may
encompass various browser technologies with different back-
end support running on various mobile OS. To ensure a reliable
service, providers have to test their services on all these
platforms.
Test automation topic is frequently visited when software
testing is considered over the cloud. There are many test
automation tools in the market, which address different re-
quirements in a testing life-cycle (e.g., automated test data
generation, test case generation, test execution and test eval-
uation). We believe our evaluation will also motivate the
migration of those tools to the cloud.
One of the major drivers of cloud computing adoption is
economies of scale. It provides a pay-per-use type of service,
thus eliminating the upfront investment in many cases. Testing
tools and services are no exception. Development teams can
benefit from this paradigm for utilizing test tools when they
need it and as much as they need it, thus saving license fees.
We will enrich the discussion with current state-of-the-art
software testing as a service over the cloud; the survey will
classify related literature according to what type of testing
activities these services support for what type of application
domains.
The remainder of the paper is organized as follows; in
Section II, we give a short background on cloud computing and
software testing in the cloud; in Section III we describe our
research methodology; in Section IV we present an evaluation
of review results and identify gaps as well as opportunities;
Section V presents related work; and we conclude the paper
in Section VI.
II. C
LOUD COMPUTING
Cloud computing is a relatively recent term, which basically
defines a new paradigm for service delivery in every aspect
of computing. It enables ubiquitous, convenient, on-demand
network access to a shared pool of configurable computing
resources (e.g., networks, servers, storage, applications, and
services) that can be rapidly provisioned and released with
minimal management effort or service provider interaction [4].
Cloud computing has been enabled by the developments
in virtualization, distributed computing, utility computing,
web and software services technologies [5]. It is especially
based on two key concepts. The first one is Service-Oriented
Architecture (SOA), which is the delivery of an integrated and
orchestrated suite of functions to an end-user. The functions
can be both loosely or tightly coupled. SOA enables end-
users to easily search, use and release services on-demand
and at a desired quality level. Workflows allow integration of
2012 IEEE Sixth International Conference on Software Security and Reliability Companion
978-0-7695-4743-5/12 $26.00 © 2012 IEEE
DOI 10.1109/SERE-C.2012.32
18