xx
grasp on to proceed in your deep-learning career. The goal of this book is not to give
you a mathematical foundation. I assume you have one. Deep learning and neural
networks (in general, machine learning) are complex, and whoever tries to convince you
otherwise is lying or doesn’t understand them.
I will not spend time in justifying or deriving algorithms or equations. You will have
to trust me there. Additionally, I will not discuss the applicability of specific equations.
For those of you with a good understanding of calculus, for example, I will not discuss
the problem of the differentiability of functions for which we calculate derivatives.
Simply assume that you can apply the formulas I give you. Many years of practical
implementations have shown the deep-learning community that those methods and
equations work as expected and can be used in practice. The kind of advanced topics
mentioned would require a separate book.
In Chapter 1, you will learn how to set up your Python environment and what
computational graphs are. I will discuss some basic examples of mathematical
calculations performed using TensorFlow. In Chapter 2, we will look at what you can
do with a single neuron. I will cover what an activation function is and what the most
used types, such as sigmoid, ReLU, or tanh, are. I will show you how gradient descent
works and how to implement logistic and linear regression with a single neuron and
TensorFlow. In Chapter 3, we will look at fully connected networks. I will discuss matrix
dimensions, what overfitting is, and introduce you to the Zalando dataset. We will
then build our first real network with TensorFlow and start looking at more complex
variations of gradient descent algorithms, such as mini-batch gradient descent. We will
also look at different ways of weight initialization and how to compare different network
architectures. In Chapter 4, we will look at dynamic learning rate decay algorithms,
such as staircase, step, or exponential decay, then I will discuss advanced optimizers,
such as Momentum, RMSProp, and Adam. I will also give you some hints on how to
develop custom optimizers with TensorFlow. In Chapter 5, I will discuss regularization,
including such well-known methods as l
1
,l
2
, dropout, and early stopping. We will look at
the mathematics behind these methods and how to implement them in TensorFlow. In
Chapter 6, we will look at such concepts as human-level performance and Bayes error.
Next, I will introduce a metric analysis workflow that will allow you to identify problems
having to do with your dataset. Additionally, we will look at k-fold cross-validation as a
tool to validate your results. In Chapter 7, we will look at the black box class of problems
and what hyperparameter tuning is. We will look at such algorithms as grid and random
search and at which is more efficient and why. Then we will look at some tricks, such
inTRoduCTion