![](https://csdnimg.cn/release/download_crawler_static/34468595/bg5.jpg)
5
Fig. 5. The architecture of Inception block [10]. Figure is reproduced
based on GoogLeNet [10].
3.1.5 DenseNet
Fig. 6. The architecture of the Dense Block proposed in DenseNet [12].
Figure is reproduced based on [12].
Most of the networks including AlexNet, VGG, and
ResNet follow a hierarchy architecture. The images are fed
to the network and features are extracted by different layers.
The shallow layers extract low-level general features, while
the deep layers extract high-level task-specific features [52].
However, when a network goes deeper, the deeper layers
may suffer from memorizing the low-level features needed
by the network to accomplish the task.
To alleviate this problem, Huang et al. proposed the
dense connection to send all the features before a convolu-
tion block as the input to the next convolution block in the
neural network [12]. As shown in Fig. 6, the output features
of all the previous convolution blocks serve as the input to
the current block. In this way, the shallower blocks focus on
the low-level general features while the deeper blocks can
focus on the high-level task-specific features.
3.2 Architectures for Learning Video Features
To extract both spatial and temporal information from
videos, several architectures have been designed for video
feature learning including 2DConvNet-based methods [53],
3DConvNet-based methods [16], and LSTM-based methods
[54]. The 2DConvNet-based methods apply 2DConvNet
on every single frame and the image features of multiple
frames are fused as video features. The 3DConvNet-based
methods employ 3D convolution operation to simultane-
ously extract both spatial and temporal features from mul-
tiple frames. The LSTM-based methods employ LSTM to
model long term dynamics within a video. This section
briefly summarizes these three types of architectures of
video feature learning.
3.2.1 Two-Stream Network
Models
RGB Frame
Fusion
Optical Flow
Inputs
Fig. 7. The general architecture of the two-stream network which includ-
ing one spatial stream and one temporal stream. Figure is reproduced
based on [53].
Videos generally are composed of various numbers of
frames. To recognize actions in a video, networks are re-
quired to capture appearance features as well as tempo-
ral dynamics from frame sequences. As shown in Fig. 7,
a two-stream 2DConvNet-based network is proposed by
Simonyan and Zisserman for human action recognition,
while using a 2DConvNet to capture spatial features from
RGB stream and another 2DConvNet to capture temporal
features from optical flow stream [53]. Optical flow encodes
boundary of moving objects, therefore, the temporal stream
ConvNet is relatively easier to capture the motion informa-
tion within the frames.
Experiments showed that the fusion of the two streams
can significantly improve action recognition accuracy. Later,
this work has been extended to multi-stream network [55],
[56], [57], [58], [59] to fuse features from different types of
inputs such as dynamic images [60] and difference of frames
[61].
3.2.2 Spatiotemporal Convolutional Neural Network
3D convolution operation was first proposed in 3DNet
[62] for human action recognition. Compared to 2DCon-
vNets which individually extract the spatial information of
each frame and then fuse them together as video features,
3DConvNets are able to simultaneously extract both spatial
and temporal features from multiple frames.
C3D [16] is a VGG-like 11-layer 3DConvNet designed for
human action recognition. The network contains 8 convolu-
tional layers, and 3 fully connected layers. All the kernels
have the size of 3 × 3 × 3, the convolution stride is fixed
to 1 pixel. Due to its powerful ability of simultaneously
extracting both spatial and temporal features from multiple
frames, the network achieved state-of-the-art on several
video analysis tasks including human action recognition
[63], action similarity labeling [64], scene classification [65],
and object recognition in videos [66].
The input of C3D is 16 consecutive RGB frames where
the appearance and temporal cues from 16-frame clips are
extracted. However, the paper of long-term temporal convo-
lutions (LTC) [67] argues that, for the long-lasting actions, 16
frames are insufficient to represent whole actions which last
longer. Therefore, larger numbers of frames are employed