![](https://csdnimg.cn/release/download_crawler_static/16479186/bg4.jpg)
Water 2018, 10, 1543 4 of 16
where
x
i
is the input value to node
i
,
O
k
is the output at node
k
,
g
1
is activation function (nonlinear)
for the hidden layer and
g
2
is activation function (linear) for the output layer.
N
and
M
represent
the number of neurons in the input and hidden layers, respectively.
W
jo
and
W
ko
are biases of the
j
th
neuron in the hidden layer and the
k
th neuron in the output layer.
W
ji
is the weight between the input
node i and the hidden node j, and W
kj
the weight between the hidden node j and the output node k.
x
1
(p)
y
1
(p)
y
2
(p)
z(p)
y
j
(p)
y
nH
(p)
x
2
(p)
x
i
(p)
x
nl
(p)
Input Layer Output Layer
Input Variables
Output Variable
Hidden Layer
n
H
i
j
n
l
x
1
(p)
x
2
(p)
x
i
(p)
x
nl
(p)
y
j
(p)
∑
2
2
1
1
Figure 1.
ANN architecture with one hidden layer (typical three-layer feed forward artificial neural
networks) [10].
2.2. RNN
Recurrent neural networks (RNNs) are powerful model for sequential data. Recurrent neural
network are a strict superset of feedforward neural networks, augmented by the inclusion of recurrent
edges that span adjacent time steps, introducing a notion of time to the model [
19
]. While RNNs
may not contain cycles among the conventional edges, recurrent edges may form cycles, including
self-connections. At time
t
, nodes receiving input along recurrent edges receive input activation from
the current example
x
t
and also from hidden nodes
h
t−1
in the network’s previous state. The output
ˆ
y
t
is calculated given the hidden state
h
t
at that time step. Thus, input
x
t−1
at time
t −
1 can influence the
output
ˆ
y
t
at time t by way of these recurrent connections (Figure 2).
We can show in two equations that all calculations are necessary for computation at each time
step on the forward pass in a simple recurrent neural network:
h
(t)
= σ(W
hx
x + W
hh
h
(t−1)
+ b
h
) (2)
ˆ
y
(t)
= so f tmax(W
gh
h
(t)
+ b
y
) (3)
where
W
hx
is the matrix of weights between the input and hidden layers and
W
hh
is the matrix of
recurrent weights between the hidden layers at adjacent time steps. The vectors
b
h
and
b
y
are biases
which allow each node to learn an offset.