14 2 Supervised learning
terms on the right-hand side of equation 2.5) and the cost function is the overall quantity
that is minimized (
i.e.
, the entire right-hand side of equation 2.5). A cost function
can contain additional terms that are not associated with individual data points. More
generally, an objective function is any function that is to be maximized or minimized.
Generative vs. discriminative models: The models y = f[x, ϕ] in this chapter are
discriminative models. These make an output prediction y from real-world measurements
x. Another approach is to build a generative model x = g[y, ϕ], in which the real-world
Problem 2.3
measurements x are computed as a function of the output y.
The generative approach has the disadvantage that it doesn’t directly predict y. To
perform inference, we must invert the generative equation as y = g
−1
[x, ϕ] and this may
be dicult. However, generative models have the advantage that we can build in prior
knowledge about how the data were generated. For example, if we wanted to predict the
3D position and orientation y of a car in an image x, then we could build knowledge
about car shape, 3D geometry, and light transport into the function x = g[y, ϕ].
This seems like a good idea, but in fact, discriminative models dominate modern machine
learning; any advantage gained from exploiting prior knowledge in generative models is
usually trumped by brute force learning of very exible discriminative models from large
amounts of training data.
Problems
Problem 2.1 To walk ‘downhill’ on the loss function (equation 2.5), we measure its slope
with respect to the parameters ϕ
0
and ϕ
1
. Calculate expressions for the slopes ∂L/∂ϕ
0
and ∂L/∂ϕ
1
.
Problem 2.2 Show that we can nd the minimum of the loss function in closed form by
setting the expression for the derivatives from problem 2.1 to zero and solving for ϕ
0
and
ϕ
1
. Note that this works for linear regression but not for more complex models; this is
why we use iterative model tting methods like gradient descent (gure 2.4).
Problem 2.3 Consider reformulating linear regression as a generative model so we have
x = g[y, ϕ] = ϕ
0
+ ϕ
1
y. What is the new loss function? Find an expression for the inverse
function y = g
−1
[x, ϕ] that we would use to perform inference. Will this model make the
same predictions as the discriminative version for a given training dataset {x
i
, y
i
}?
This work is subject to a Creative Commons CC-BY-NC-ND license. (C) MIT Press.
剩余202页未读，继续阅读
