A. Sarı, A. Tosun and G.I. Alptekin / The Journal of Systems and Software 153 (2019) 200–219 203
Table 2
Research questions.
Research question Aim
RQ1: What are the business models used for crowdsourcing in software
engineering (CSE)?
Analyzing the business models used for CSE.
RQ2: What are the technological platforms used for management of CSE? Investigating technical infrastructure on which crowdsourcing process is
implemented
RQ3: How are crowdsourced software development processes modeled? Identifying crowdsourced software
development methodologies
RQ4: For which software process area(s) crowdsourcing is utilized? Identifying software process area(s) that crowdsourcing is utilized
RQ5: What kind of effort estimation approaches are employed in
crowdsourced software development?
Identifying effort estimation approaches in crowdsourced software
development
RQ6: What are the cost drivers used in effort estimation?
Identifying the factors that affect effort estimation
RQ7: How are task awards determined in CSE? Investigating task award strategies in CSE
RQ8: What kind of strategies exist for crowd selection or formation in
software engineering?
Analyzing strategies for crowd selection or formation in software engineering
RQ9: How are tasks decomposed into
micro-tasking process performed in CSE? Investigating micro-tasking process performed in CSE
RQ10: Which tools have been used to assist CSE? Identifying assisting tools for CSE
decision to crowdsource. The authors broadly classified main crite-
ria that affect the decision to crowdsource as: task, people, man-
agement and environmental issues. They proposed a theoretical
framework for deciding whether to crowdsource or not. They also
suggest mechanisms to increase commitment of internal employ-
ees in software companies, and pointed out the importance of
available platforms for crowdsourcing, and risks of achieving low
quality results and loss of intellectual property. In Stol et al. (2017) ,
the authors have only reviewed related articles instead of conduct-
ing a systematic review. The authors selected six articles out of 18
submissions to the crowdsourcing call to demonstrate how soft-
ware development can benefit from crowdsourcing as a source of
knowledge, and as a source for new ideas and feedbacks on exist-
ing software. A survey in the context of gamification and crowd-
sourcing is generated in Morschheuser et al. (2017) . They stated
that gamification was used to promote a kind of competition be-
tween the participants rather than a collaborative experience. They
explored that in most of the analyzed cases, a supplementary fi-
nancial incentive (monetary reward) was not involved. Moreover,
the research examined structured conceptual framework of gami-
fied crowdsourcing systems and psychological and behavioral out-
comes associated with the use of gamification affordance. Although
the paper examines gamification with the crowdsourcing perspec-
tive and by using crowdsourcing terminology, its findings are all in
the gamification field.
As pointed out in each literature survey, the number of CSE-
related works has been expanding every day. Our SLR covers pub-
lications before January 2018, and synthesizes evidence from 67
primary studies that passed our quality assessment criteria. Except
Mao et al. (2017) , none of the earlier works conducted a SLR on
this topic. Hence, we have noticed that there is a need to conduct
a more comprehensive overview to identify and aggregate exist-
ing evidence in CSE, by putting more emphasis on unexplored is-
sues. In the most comprehensive SLR related to CSE ( Mao et al.,
2017 ), the authors state CSE theories and models, task decomposi-
tion and remuneration are among the open issues in crowdsourc-
ing applications in the field of software engineering. Hence, we
introduced new research questions, namely RQ1, RQ2, RQ5, RQ6,
RQ7, RQ9. Although the tools that assist CSE are studied in detail
by Mao et al. (2017) , in our research we associate the assisted tools
with assisted task, award mechanism and supported crowd selec-
tion strategy. Our work can be considered as one of the pioneering
works that deeply examine the economic impact of CSE in terms of
effort estimation models, remuneration/t ask awarding models and
cost drivers, and the crowd formation strategies.
3. Research methodology
We used SLR guidelines proposed by Kitchenham and Char-
ters (2007) conducting this study. SLR steps were followed by
defining research questions, search string and databases, exploring
relevant papers, running a two-phase filtering method via quality
assessment checklists, generating the final list of primary studies
and then synthesizing useful data through thematic analysis. This
section reports the details of our SLR steps.
3.1. Definition of research questions
We list our 10 research questions (RQs) to synthesize and ag-
gregate the existing evidence in CSE and to identify emerging re-
search and practical issues in Table 2 .
3.2. Conducting the research
A keyword search is a type of search that looks for match-
ing documents involving one or more words specified by the user.
These words need to distinguish the most relevant research among
scientific databases. We defined several keywords and formulated
them via logical operators:
•
(crowdsourcing OR crowdsourced OR crowd) AND "software en-
gineering"
•
(crowdsourcing OR crowdsourced OR crowd) AND "software de-
velopment"
•
"competitive programming"
•
"crowdsourced development"
Although there are various other databases, we chose six pop-
ular ones, which have the richest publication content in our re-
search field: IEEEXplore, ACM Digital Library, Web of Science, SCO-
PUS, SpringerLink and ScienceDirect. These databases were also
used in the earlier secondary studies ( Ambreen and Naveed, 2016;
Hosseini et al., 2015; Mao et al., 2017; Morschheuser et al., 2017;
Stol et al., 2017; Thuan et al., 2016 ). The searching was performed
by using our keywords and extracting all published work from
these libraries. Our searching process was carried out without any
date limit. We did not include Google Scholar in order to limit our
search domain to the publications only. In crowdsourcing, we are
aware that there are many blog posts and newspaper and mag-
azine articles that target software practitioners. However, in this
SLR we focus on academic publications only. Table 3 summarizes
the full-text search execution in these databases in terms of num-
ber of publications filtered at each phase. Initial search revealed
5574 publications in total.
3.3. Screening of relevant papers
The screening process is comprised of establishing inclusion/
exclusion criteria in order to find relevant studies regarding to our
RQs, and an additional quality assessment to select the final list of
primary studies.