5
Fig. 3: Word Cloud based on the Venue Names of Selected
Primary Publications.
1
4
9
15
27
23
24
2010 2011 2012 2013 2014 2015 2016
Fig. 4: The number of publications in each year.
Table A1 (in appendix) enumerates the details of those 103
publications.
It is noteworthy that around 4% of the final primary
publications are exclusively found by major venues search,
meaning that they cannot be found based on well-known
online repositories such as IEEE and ACM. This result, along
with our previous experiences [22], suggests that repository
search is necessary but not sufficient for harvesting review
publications. Other steps (e.g., top venues search based on
Google Scholar impact factor [22] or CCF ranking) should
be taken in complement to ensure reliable coverage of state-
of-the-art papers.
Fig. 3 presents a word cloud based on the venue names
of selected primary publications. The more papers selected
from a venue, the bigger its name showing in the word
cloud. Not surprisingly, the recurrently targeted venues
are mainly testing-related conferences such as ISSTA, ICST,
ISSRE, etc.
Fig. 4 illustrates the trend of the number of publications
in each year we have considered. From this figure, we can
observe that the number of papers tackling the problem
of testing Android apps has increased gradually to reach
a peak in 2014. Afterwards, the pace of developing new
testing techniques has stabilized.
We further look into the selected primary publications
through their published venue types and domains. Fig. 5a
and Fig. 5b illustrate the statistic results, respectively. Over
90% of examined papers are published in conferences and
workshops (which are usually co-located with top confer-
ences) while only 10% papers are published in journals.
These findings are in line with the current situation where
intense competition in Android research forces researchers
70.9%
9.7%
19.4%
Conference Journal/Magazine Wor k sh op
(a) Venue Types.
82.5%
6.8%
10.7%
SE/PL Other SEC
(b) Venue Domains.
Fig. 5: Distribution of examined publications through published
venue types and domains.
to make available their works as fast as possible. We further
find that over 80% of examined papers are published in
software engineering and programming language venues,
showing that testing Android apps is mainly a concern
in the software engineering community. Nevertheless, as
shown by several papers published in proceedings of se-
curity venues, testing is also a valuable approach to address
security issues in Android apps.
4 TAXONOMY OF ANDROID TESTING RESEARCH
To extract relevant information from the literature, our SLR
must focus on specific characteristics eventually described
in each publication. To facilitate this process in a field that
explores a large variety of approaches, we propose to build
a taxonomy of Android testing. Such a taxonomy eventually
helps to gain insights into the state-of-the-art by answering
the research questions proposed in Section 2.1.
By searching for answers to the aforementioned research
questions in each publication, we are able to make a system-
atic assessment of the literature with a schema for classify-
ing and comparing different approaches. Fig. 6 presents a
high-level view of the taxonomy diagram spreading in four
dimensions (i.e., Test Objectives, Test Targets, Test Levels
and Test Techniques) associated with the first three research
questions
13
.
Test Objectives. This dimension summarizes the tar-
geted objectives of our examined testing-related publica-
tions. We have enumerated overall 6 recurring testing ob-
jectives such as Bug/Defect detection.
Test Targets. This dimension summarizes the representa-
tive targets where testing approaches focus on. In particular,
for testing Android apps, the GUI/Event and ICC/IAC are
recurrently targeted. For simplicity, we regroup all the other
targets such as normal code analysis into General.
Test Levels. This dimension checks the different levels
(also known as phases) at which the test activities are per-
formed. Indeed, there is a common knowledge that software
testing is very important and has to be applied to many
levels such as unit testing, integration testing, etc. Android
apps, as a specific type of software, also need to go through
13. Test Objectives and Test Targets for RQ1 (test concerns), Test
Levels for RQ2 (test levels) and Test Techniques for RQ3 (test ap-
proaches). RQ4 explores the validity of testing approaches that is not
summarised in the taxonomy.