4. We believe we are the first to show two compelling
findings: (i) the number of developers modifying a file
is inversely correlated to the quality of software in that
file, and (ii) the software changes pushed on the day
of the release cut from the Master branch are of lower
quality than the files pushed on other days.
In the next section we describe the mobile release cycle
used at FB for both Android and iOS. Section 3 presents
the collected data that we used for our analysis. Section 4
describes the testing strategies used. Our analysis is pre-
sented in §5. Related work is covered in §6. We close with
concluding remarks.
2. MOBILE RELEASE CYCLE
In this section we describe in detail the development and
deployment processes and activities at FB for mobile apps.
The overall architecture is shown in Figure 1. As seen in
the figure, there are two classes of activities: development
activities and deployment activities.
2.1 Development Activities
First we describe development activities. There is effectively
no difference between the development activities for mobile-
based and cloud-based software at FB. The developer forks a
local revision control system branch from the Master branch.
Updates to the software are made to the local branch with
frequent commits. After proper testing and when the devel-
oper believes her software update is ready for deployment,
she pushes
2
the updates into the Master branch. As we show
in §5, each developer pushes such updates into the Master
branch 3-5 times a week, a timescale comparable to cloud-
based software at FB.
A notable coding practice encouraged at FB is the use of
a mechanism called Gatekeeper, that allows one to dynami-
cally control the availability of features on the mobile device.
In effect, this mechanism provides the ability to dynamically
turn on or off a mobile-side feature from the server side,
even on devices in client hands. Hence, if a newly deployed
feature misbehaves, it can be turned off. This mechanism
can also be used to incrementally turn on new features in a
targeted way, say by targeting the OS, the specific OS ver-
sion, the hardware device type, the specific hardware device
model, the country, locale, etc. It can also be used for A/B
testing, for example to test whether one feature gets more
usage than an alternative.
2.2 Deployment Activities
We now describe the lower half of the figure correspond-
ing to deployment activities. FB has a Release Engineering
Team (RelEng) that is responsible for these activities. The
team includes project managers who are responsible for their
target platform and who can make informed decisions as to
which issues are launch-blocking or not.
Both iOS and Android deploy software on a fixed-date
cycle in a pipelined fashion. We begin by describing the
specific process for iOS and then describe how the process
for Android differs.
2
In this paper, we use the term push exclusively to refer
to the pushing of the local development branch up into the
Master branch; we do not use the term here to refer to the
act of deploying software.
Deployment
Development
2"weeks"of"development
2"weeks"of"development
stabilizing soak
Master branch
Release
branch
3 daily dog-food
builds
deployment
cherry
-picks
pushes pushes
Figure 1: Release cycle for iOS. The process is pipelined
with each stage taking two weeks (for iOS mobile code): the
stabilization, soak, review, and deployment with the Re-
lease branch takes two week, while new software updates
are continuously being pushed to the master branch during
the same two weeks until the next Release branch is cut from
the Master branch.
iOS
For iOS, a Release branch is cut from the Master branch by
RelEng at a two week cadence every other Sunday at 6pm.
The release branch is first stabilized over a period of five
days, during which issues are fixed or eliminated, and other
polish and stabilization updates are applied. Some bugs are
categorized by RelEng as launch-blocking; that is, bugs that
would prevent making the app available to end-users and
hence have to be fixed before deployment.
3
Finally, RelEng
may decide to circumvent some issues by reverting parts of
the code back to a previous version.
Developers are responsible for addressing any of the issues
raised by RelEng, such as identified bugs. Each update gen-
erated by a developer that addresses issues raised by RelEng
(regarding bug fixes, polishes, and stabilization) is only ever
pushed into the Master branch. The developer must then
make a request to RelEng to merge the update from the
Master branch into the Release branch. In doing so, she
must provide justification as to why the update should be
merged into the Release branch.
In turn, RelEng “cherry-picks” (i.e. selects), at its dis-
cretion, those updates that will be merged into the Release
branch. It does this carefully by taking various risk factors
into account. Merge requests may be declined for any num-
ber of reasons. Two examples of merge requests that would
likely be declined simply because the risk is deemed to be
too high are: updates having too many dependencies (e.g.,
more than 5) or updates that make significant changes to
core components like networking or display. Ultimately, the
decision is a judgement call by RelEng involving a trade-
off between risk and impact. For example, a polish update,
which may have lower impact, is only accepted during the
first 2-3 days and declined after that.
During the above described period of stabilization, the re-
lease branch is built and the resulting app is made available
3
Note that despite the name, launch-blocking issues never
get to the point where they actually block a deployment.
14