11
Chapter 2: Test Planning
Failing to adequately plan a testing effort will frequently result in the project's sponsors
being unpleasantly surprised. The surprise could be in the form of an unexpected cost
overrun by the testing team, or finding out that a critical component of a Web site wasn't
tested and consequently permitted an intruder to gain unauthorized access to company
confidential information.
This chapter looks at the key decisions that a security-testing team needs to make while
planning their project, such as agreeing on the scope of the testing effort, assessing the
risks (and mitigating contingencies) that the project may face, spelling out any rules of
engagement (terms of reference) for interacting with a production environment, and
specifying which configuration management practices to use. Failing to acknowledge any
one of these considerations could have potentially dire consequences to the success of the
testing effort and should therefore be addressed as early as possible in the project. Black
(2002), Craig et al. (2002), Gerrard et al. (2002), Kaner et al. (1999, 2001), the Ideahamster
Organization (www.ideahamster.org), and the Rational Unified Process (www.rational.com)
provide additional information on planning a testing project.
Requirements
A common practice among testing teams charged with evaluating how closely a system will
meet its user's (or owner's) expectations is to design a set of tests that confirm whether or
not all of the features explicitly documented in a system's requirements specification have
been implemented correctly. In other words, the objectives of the testing effort are
dependent upon on the system's stated requirements. For example, if the system is
required to do 10 things and the testing team runs a series of tests that confirm that the
system can indeed accurately perform all 10 desired tasks, then the system will typically be
considered to have passed. Unfortunately, as the following sections seek to illustrate, this
process is nowhere near as simple a task to accomplish as the previous statement would
lead you to believe.
Clarifying Requirements
Ideally, a system's requirements should be clearly and explicitly documented in order for the
system to be evaluated to determine how closely it matches the expectations of the
system's users and owners (as enshrined by the requirements documentation).
Unfortunately, a testing team rarely inherits a comprehensive, unambiguous set of
requirements; often the requirements team-or their surrogates, who in some instances may
end up being the testing team-ends up having to clarify these requirements before the
testing effort can be completed (or in some cases started). The following are just a few
situations that may necessitate revisiting the system's requirements:
Implied requirements. Sometimes requirements are so obvious (to the
requirements author) that the documentation of these requirements is deemed to be a
waste of time. For example, it's rare to see a requirement such as "no spelling
mistakes are to be permitted in the intruder response manual" explicitly documented,
but at the same time, few organizations would regard spelling mistakes as desirable.
Incomplete or ambiguous requirements. A requirement that states, "all the Web
servers should have service pack 3 installed," is ambiguous. It does not make it clear
whether the service pack relates to the operating system or to the Web service
(potentially different products) or which specific brand of system software is required.