The Matrix Calculus You Need For Deep Learning.pdf

The Matrix Calculus You Need For Deep Learning

Terence Parr and Jeremy Howard

July 3, 2018

(We teach in University of San Francisco’s MS in Data Science program and have other nefarious

projects underway. You might know Terence as the creator of the ANTLR parser generator. For

more material, see Jeremy’s fast.ai courses and University of San Francisco’s Data Institute in-

person version of the deep learning course.)

HTML version (The PDF and HTML were generated from markup using bookish)

Abstract

if you get stuck at some point along the way—just go back and reread the previous section, and

try writing down and working through some examples. And if you’re still stuck, we’re happy

to answer your questions in the Theory category at forums.fast.ai. Note: There is a reference

section at the end of the paper summarizing all the key matrix calculus rules and terminology

discussed here.

1

arXiv:1802.01528v3 [cs.LG] 2 Jul 2018

Contents

1 Introduction 3

2 Review: Scalar derivative rules 4

5 The gradient of neuron activation 23

6 The gradient of the neural network loss function 25

7 Summary 29

8 Matrix Calculus Reference 29

10 Resources 32

2

1 Introduction

Most of us last saw calculus in school, but derivatives are a critical part of machine learning,

particularly deep neural networks, which are trained by optimizing a loss function. Pick up a

machine learning paper or the documentation of a library such as PyTorch and calculus comes

screeching back into your life like distant relatives around the holidays. And it’s not just any old

scalar calculus that pops up—you need differential matrix calculus, the shotgun wedding of linear

algebra and multivariate calculus.

Well... maybe need isn’t the right word; Jeremy’s courses show how to become a world-class deep

learning practitioner with only a minimal level of scalar calculus, thanks to leveraging the automatic

differentiation built in to modern deep learning libraries. But if you really want to really understand

function and is followed by a rectified linear unit, which clips negative values to zero: max(0, z(x)).

Such a computational unit is sometimes referred to as an “artificial neuron” and looks like:

Neural networks consist of many of these units, organized into multiple collections of neurons called

layers. The activation of one layer’s units become the input to the next layer’s units. The activation

of the unit or units in the final layer is called the network output.

Training this neuron means choosing weights w and bias b so that we get the desired output for all N

If we’re careful, we can derive the gradient by differentiating the scalar version of a common loss

3

function (mean squared error):

1

N

X

(target(x) − activation(x))

=

X

w

x

+ b))

matrix calculus, and the good news is, we only need a small subset of that field, which we introduce

here. While there is a lot of online material on multivariate calculus and linear algebra, they are

typically taught as two separate undergraduate courses so most material treats them in isolation.

The pages that do discuss matrix calculus often are really just lists of rules with minimal explanation

or are just pieces of the story. They also tend to be quite obscure to all but a narrow audience

of mathematicians, thanks to their use of dense notation and minimal discussion of foundational

concepts. (See the annotated list of resources at the end.)

In contrast, we’re going to rederive and rediscover some key matrix calculus rules in an effort to

explain them. It turns out that matrix calculus is really not that hard! There aren’t dozens of new

rules to learn; just a couple of key concepts. Our hope is that this short paper will get you started

calculus to become an effective practitioner.)

A note on notation: Jeremy’s course exclusively uses code, instead of math notation, to explain

concepts since unfamiliar functions in code are easy to search for and experiment with. In this

paper, we do the opposite: there is a lot of math notation because one of the goals of this paper is

to help you understand the notation that you’ll see in deep learning papers and books. At the end

of the paper, you’ll find a brief table of the notation used, including a word or phrase you can use

to search for more details.

2 Review: Scalar derivative rules

Hopefully you remember some of these main scalar derivative rules. If your memory is a bit fuzzy

on this, have a look at Khan academy vid on scalar derivative rules.

4

Power Rule x

= 3x

Sum Rule f + g

+

+ 3x) = 2x + 3

Product Rule fg f

+

x = x

Chain Rule f(g(x))

) =

2x =

5