A STATISTICAL TEST SUITE FOR RANDOM AND PSEUDORANDOM NUMBER GENERATORS FOR CRYPTOGRAPHIC APPLICATIONS
The probability of a Type I error is often called the level of significance of the test. This probability can
be set prior to a test and is denoted as α. For the test, α is the probability that the test will indicate that the
sequence is not random when it really is random. That is, a sequence appears to have non-random
properties even when a “good” generator produced the sequence. Common values of α in cryptography
are about 0.01.
The probability of a Type II error is denoted as β. For the test, β is the probability that the test will
indicate that the sequence is random when it is not; that is, a “bad” generator produced a sequence that
appears to have random properties. Unlike α, β is not a fixed value. β can take on many different values
because there are an infinite number of ways that a data stream can be non-random, and each different
way yields a different β. The calculation of the Type II error β is more difficult than the calculation of α
because of the many possible types of non-randomness.
One of the primary goals of the following tests is to minimize the probability of a Type II error, i.e., to
minimize the probability of accepting a sequence being produced by a generator as good when the
generator was actually bad. The probabilities α and β are related to each other and to the size n of the
tested sequence in such a way that if two of them are specified, the third value is automatically
determined. Practitioners usually select a sample size n and a value for α (the probability of a Type I
error – the level of significance). Then a critical point for a given statistic is selected that will produce the
smallest β (the probability of a Type II error). That is, a suitable sample size is selected along with an
acceptable probability of deciding that a bad generator has produced the sequence when it really is
random. Then the cutoff point for acceptability is chosen such that the probability of falsely accepting a
sequence as random has the smallest possible value.
Each test is based on a calculated test statistic value, which is a function of the data. If the test statistic
value is
S and the critical value is t, then the Type I error probability is P(S > t || H
o
is true) = P(reject H
o
|
H
0
is true), and the Type II error probability is P(S ≤ t || H
0
is false) = P(accept H
0
| H
0
is false). The test
statistic is used to calculate a P-value that summarizes the strength of the evidence against the null
hypothesis. For these tests, each P-value is the probability that a perfect random number generator would
have produced a sequence less random than the sequence that was tested, given the kind of non-
randomness assessed by the test. If a P-value for a test is determined to be equal to 1, then the sequence
appears to have perfect randomness. A P-value of zero indicates that the sequence appears to be
completely non-random. A significance level (
α
) can be chosen for the tests. If P-value
≥ α
, then the
null hypothesis is accepted; i.e., the sequence appears to be random. If P-value
< α
, then the null
hypothesis is rejected; i.e., the sequence appears to be non-random. The parameter
α
denotes the
probability of the Type I error. Typically,
α
is chosen in the range [0.001, 0.01].
• An α of 0.001 indicates that one would expect one sequence in 1000 sequences to be rejected by
the test if the sequence was random. For a P-value ≥ 0.001, a sequence would be considered to be
random with a confidence of 99.9%. For a P-value < 0.001, a sequence would be considered to be non
-
random with a confidence of 99.9%.
• An α of 0.01 indicates that one would expect 1 sequence in 100 sequences to be rejected. A P-
value ≥ 0.01 would mean that the sequence would be considered to be random with a confidence of 99%.
A P-value < 0.01 would mean that the conclusion was that the sequence is non-random with a confidence
of 99%.
For the examples in this document, α has been chosen to be 0.01. Note that, in many cases, the
parameters in the examples do not conform to the recommended values; the examples are for illustrative
purposes only.