ii TABLE OF CONTENTS
3. PROBABILITY, STATISTICS AND INFORMATION THEORY..73
3.1. P
ROBABILITY THEORY .........................................................................................74
3.1.1. Conditional Probability And Bayes' Rule............................................75
3.1.2. Random Variables................................................................................77
3.1.3. Mean and Variance..............................................................................79
3.1.4. Covariance and Correlation................................................................83
3.1.5. Random Vectors and Multivariate Distributions.................................84
3.1.6. Some Useful Distributions ...................................................................85
3.1.7. Gaussian Distributions ........................................................................92
3.2. E
STIMATION THEORY ...........................................................................................98
3.2.1. Minimum/Least Mean Squared Error Estimation................................99
3.2.2. Maximum Likelihood Estimation .......................................................104
3.2.3. Bayesian Estimation and MAP Estimation ........................................108
3.3. S
IGNIFICANCE TESTING.......................................................................................114
3.3.1. Level of Significance..........................................................................114
3.3.2. Normal Test (Z-Test)..........................................................................116
3.3.3.
2
Goodness-of-Fit Test...................................................................117
3.3.4. Matched-Pairs Test............................................................................119
3.4. I
NFORMATION THEORY ......................................................................................121
3.4.1. Entropy ..............................................................................................121
3.4.2. Conditional Entropy ..........................................................................124
3.4.3. The Source Coding Theorem .............................................................125
3.4.4. Mutual Information and Channel Coding..........................................127
3.5. H
ISTORICAL PERSPECTIVE AND FURTHER READING...........................................129
4. PATTERN RECOGNITION...........................................................................133
4.1. B
AYES DECISION THEORY..................................................................................134
4.1.1. Minimum-Error-Rate Decision Rules ................................................135
4.1.2. Discriminant Functions .....................................................................138
4.2. H
OW TO CONSTRUCT CLASSIFIERS.....................................................................140
4.2.1. Gaussian Classifiers ..........................................................................142
4.2.2. The Curse of Dimensionality..............................................................144
4.2.3. Estimating the Error Rate..................................................................146
4.2.4. Comparing Classifiers .......................................................................148
4.3. D
ISCRIMINATIVE TRAINING ................................................................................150
4.3.1. Maximum Mutual Information Estimation.........................................150
4.3.2. Minimum-Error-Rate Estimation.......................................................156
4.3.3. Neural Networks ................................................................................158
4.4. U
NSUPERVISED ESTIMATION METHODS .............................................................163
4.4.1. Vector Quantization...........................................................................164
4.4.2. The EM Algorithm .............................................................................170
4.4.3. Multivariate Gaussian Mixture Density Estimation...........................172
评论3