逻辑回归和最大熵的等价性证明

The equivalence of logistic regression and maximum entropy

models

John Mount

∗

September 23, 2011

Abstract

As our colleague so aptly demonstrated ( http://www.win-vector.com/blog/2011/09/the-simpler-

derivation-of-logistic-regression/ (link) ) there is one derivation of Logistic Regression that is particularly

beautiful. It is not as general as that found in Agresti[Agresti, 1990] (which deals with generalized

not just one of many equivalent GLMs).

1 Overview

We will proceed as follows:

1. This outline.

2. Introduce a simplified machine learning problem and some notation.

3. Re-discover logistic regression by a simplified version of the standard derivations.

4. Re-invent logistic regression by using the maximum entropy method.

5. Draw some conclusions.

2 Notation

Suppose our machine learning input is a sequence of real vectors of dimension n (where we have pre-

email: mailto:jmount@win-vector.com web: http://www.win-vector.com/