(Image by Author)
It learns the network’s parameters (weights) such that it can output the Optimal Q values.
The underlying principle of a Deep Q Network is very similar to the Q Learning algorithm. It starts with
arbitrary Q-value estimates and explores the environment using the
ε
-greedy policy. And at its core, it uses
the same notion of dual actions, a current action with a current Q-value and a target action with a target Q-
value, for its update logic to improve its Q-value estimates.
DQN Architecture Components
The DQN architecture has two neural nets, the Q network and the Target networks, and a component called
Experience Replay. The Q network is the agent that is trained to produce the Optimal State-Action value.
Experience Replay interacts with the environment to generate data to train the Q Network.
(Image by Author)
The Q Network is a fairly standard neural network architecture and could be as simple as a linear network
with a couple of hidden layers if your state can be represented via a set of numeric variables. Or if your state
data is represented as images or text, you might use a regular CNN or RNN architecture.
The Target network is identical to the Q network.