![](https://csdnimg.cn/release/download_crawler_static/11188362/bg14.jpg)
Dive into Deep Learning, Release 0.7
3.2 The Key Components: Data, Models, and Algorithms
In our wake-word example, we described a dataset consisting of audio snippets and binary labels gave a hand-wavy sense
of how we might train a model to approximate a mapping from snippets to classifications. This sort of problem, where we
try to predict a designated unknown label given known inputs (also called features or covariates), and examples of both
is called supervised learning, and it’s just one among many kinds of machine learning problems. In the next section, we’ll
take a deep dive into the different ML problems. First, we’d like to shed more light on some core components that will
follow us around, no matter what kind of ML problem we take on:
1. The data that we can learn from
2. A model of how to transform the data
3. A loss function that quantifies the badness of our model
4. An algorithm to adjust the model’s parameters to minimize the loss
3.2.1 Data
It might go without saying that you cannot do data science without data. We could lose hundreds of pages pondering the
precise nature of data but for now we’ll err on the practical side and focus on the key properties to be concerned with.
Generally we are concerned with a collection of examples (also called data points, samples, or instances). In order to work
with data usefully, we typically need to come up with a suitable numerical representation. Each example typically consists
of a collection of numerical attributes called features or covariates.
If we were working with image data, each individual photograph might constitute an example, each represented by an
ordered list of numerical values corresponding to the brightness of each pixel. A 200 × 200 color photograph would
consist of 200 × 200 × 3 = 120000 numerical values, corresponding to the brightness of the red, green, and blue
channels corresponding to each spatial location. In a more traditional task, we might try to predict whether or not a
patient will survive, given a standard set of features such as age, vital signs, diagnoses, etc.
When every example is characterized by the same number of numerical values, we say that the data consists of fixed-length
vectors and we describe the (constant) length of the vectors as the dimensionality of the data. As you might imagine, fixed
length can be a convenient property. If we wanted to train a model to recognize cancer in microscopy images, fixed-length
inputs means we have one less thing to worry about.
However, not all data can easily be represented as fixed length vectors. While we might expect microscrope images to
come from standard equipment, we can’t expect images mined from the internet to all show up in the same size. While
we might imagine cropping images to a standard size, text data resists fixed-length representations even more stubbornly.
Consider the product reviews left on e-commerce sites like Amazon or TripAdvisor. Some are short: “it stinks!”. Others
ramble for pages. One major advantage of deep learning over traditional methods is the comparative grace with which
modern models can handle varying-length data.
Generally, the more data we have, the easier our job becomes. When we have more data, we can train more powerful
models, and rely less heavily on pre-conceived assumptions. The regime change from (comparatively small) to big data is
a major contributor to the success of modern deep learning. To drive the point home, many of the most exciting models
in deep learning either don’t work without large data sets. Some others work in the low-data regime, but no better than
traditional approaches.
Finally it’s not enough to have lots of data and to process it cleverly. We need the right data. If the data is full of
mistakes, or if the chosen features are not predictive of the target quantity of interest, learning is going to fail. The
situation is well captured by the cliché: garbage in, garbage out. Moreover, poor predictive performance isn’t the only
potential consequence. In sensitive applications of machine learning, like predictive policing, resumé screening, and risk
models used for lending, we must be especially alert to the consequences of garbage data. One common failure mode
occurs in datasets where some groups of people are unrepresented in the training data. Imagine applying a skin cancer
recognition system in the wild that had never seen black skin before. Failure can also occur when the data doesn’t merely
under-represent some groups, but reflects societal prejudices. For example if past hiring decisions are used to train a
14 Chapter 3. Introduction