Software Engineering at Google, by Fergus Henderson
code files containing a total of 2 billion lines of source code, with a history of 35 million
commits and a change rate of 40 thousand commits per work day [18]. Write access to the
repository is controlled: only the listed owners of each subtree of the repository can approve
changes to that subtree. But generally any engineer can access any piece of code, can check it
out and build it, can make local modifications, can test them, and can send changes for review
by the code owners, and if an owner approves, can check in (commit) those changes.
Culturally, engineers are encouraged to fix anything that they see is broken and know how to fix,
regardless of project boundaries. This empowers engineers and leads to higher-quality
infrastructure that better meets the needs of those using it.
Almost all development occurs at the “head” of the repository, not on branches. This helps
identify integration problems early and minimizes the amount of merging work needed. It also
makes it much easier and faster to push out security fixes.
Automated systems run tests frequently, often after every change to any file in the transitive
dependencies of the test, although this is not always feasible. These systems automatically
notify the author and reviewers of any change for which the tests failed, typically within a few
minutes. Most teams make the current status of their build very conspicuous by installing
prominent displays or even sculptures with color-coded lights (green for building successfully
and all tests passing, red for some tests failing, black for broken build). This helps to focus
engineers’ attention on keeping the build green. Most larger teams also have a “build cop” who
is responsible for ensuring that the tests continue to pass at head, by working with the authors
of the offending changes to quickly fix any problems or to roll back the offending change. (The
build cop role is typically rotated among the team or among its more experienced members.)
This focus on keeping the build green makes development at head practical, even for very large
teams.
Code ownership. Each subtree of the repository can have a file listing the user ids of the
“owners” of that subtree. Subdirectories also inherit owners from their parent directories,
although that can be optionally suppressed. The owners of each subtree control write access to
that subtree, as described in the code review section below. Each subtree is required to have at
least two owners, although typically there are more, especially in geographically distributed
teams. It is common for the whole team to be listed in the owners file. Changes to a subtree
can be made by anyone at Google, not just the owners, but must be approved by an owner.
This ensures that every change is reviewed by an engineer who understands the software being
modified.
For more on the source code repository at Google, see [17, 18, 21]; and for how another large
company deals with the same challenge, see [19].
2.2. The Build System
Google uses a distributed build system known as Blaze, which is responsible for compiling and
4