This thing had everything. With the click of a button, you could fire up the
app, record a test, and then play it back while making assertions about things
you had seen along the way. It was easy to use. Running the tests made for
great demos (because you could actually see the application running). And
the best part of all, our build engineers had even found a way to include the
scripts as part of our continuous build and integration process—meaning we
would know instantly if something had broken.
It was a masterpiece of human ingenuity.
We loved it. Our testers loved it. Our customers loved it. And everything was
great. Until…
The Wheels on the Bus
It didn’t happen all once. It kind of snuck up on us slowly at first. But we
eventually began to notice that the more we used our automated testing
framework, the harder it got to add new features to the system.
At first it wasn’t immediately clear why. We had good automated test coverage.
We were continuously integrating our changes and regularly releasing the
software to clients. We saw no reasons why writing lots of automated UI tests
like this should slow us down. But when we dug a little deeper, we discovered
a few disturbing trends.
First off, developers had stopped writing a certain kind of automated test
called a unit test. These tests were fast, little code-based tests that we relied
on to tell us quickly if anything was ever broken in the software. By not
writing those, and instead replacing them with longer-running user interface
tests, our automated builds took longer to run. Which meant we didn’t know
till much later which changes broke which tests.
This in turn created another problem. Because the tests were now taking
longer to run, developers stopped running them. Many started ignoring them
altogether. We had deadlines after all, and builds that used to take ten minutes
were now taking upwards of three hours. No one had time to wait for a three-
hour build. So the build was perpetually broken. And even worse, people
started checking in new code on top of it.
Then one day it all came to a head. We missed a critical deadline. We were
stuck. There were tons of bugs in the software. We couldn’t easily add any
new functionality. And for the first time, we had to confront the uncomfortable
truth that our beloved test framework was the source of many of our problems.
Chapter 1. The Testing Pyramid • 4
report erratum • discuss
