![](https://csdnimg.cn/release/download_crawler_static/16265669/bg4.jpg)
4
SCIENTIFIC REPORTS | (2019) 9:19038 | https://doi.org/10.1038/s41598-019-55320-6
www.nature.com/scientificreports
www.nature.com/scientificreports/
an unsupervised method that incorporates ReLUs and recent unsupervised regularization techniques. Second,
analysing the benets of unsupervised pre-training compared to data augmentation and dropout on a bench-
mark dataset, while varying the ratio of unsupervised to supervised samples. ird, verifying their ndings using
benchmark datasets. ey found that unsupervised pre-training is a perfect way to improve the performance,
especially when the ratio of unsupervised to supervised samples is high
34
. Tang et al. presented a new pre-training
approach based on knowledge transfer learning
35
. In contrast to the layer-wise approach, which trains the model
components layer-by-layer, their approach trains the entire model as a whole. In their experiments on speech
recognition, they could learn complex RNNs as a student model by a weaker deep neural network (DNN) as a
teacher model. Furthermore, their approach can be combined with a layer-wise pre-training of CNN to deliver
additional gains.
Recently, Saikia et al. empirically analysed the eect of dierent unsupervised pre-training approaches in
modelling regression problems using four dierent datasets
30
. ey considered two methods separately, namely,
DBN and stacked autoencoder (SA), and compared their performance with a standard ANN algorithm without
pre-training. Liu et al. applied Hinton’s model on stacked autoencoders (SAEs) to solve gearbox fault diagno-
sis
37
. e proposed method can directly extract salient features from frequency-domain signals and eliminate
the exhaustive use of hand-craed features. Meng et al. used it also in order to train the stacked denoising sparse
autoencoder layer-by-layer
38
.
As we mentioned in this section, most of the previous work in unsupervised pre-training NN (or deep NNs)
has focused on data compression
20
, dimensionality reduction
20,27
, classication
20,28
, and UTS forecasting
20
prob-
lems. Importantly, time series forecasting with deep learning techniques is an interesting research area that needs
to be studied as well
19,26
. Moreover, even the recent time series forecasting research in the literature has focused
on UTS problems. Very few works in the literature are devoted to forecasting via MTS data. For example, Romeu
et al. used a pre-trained autoencoder-based DNN for UTS forecasting of indoor temperature
26
. Kuremoto et al.
used the earlier Hinton model for unsupervised pre-training step and backpropagation for the ne-tuning step
using an articial UTS dataset
39
. Malhotra et al. extended an approach that uses a CNN with data from audio
and image domains to be used in the time series domain and trained a multilayer RNN that can then be used as a
pre-trained model to obtain representations for time series
40
. eir method overcomes the need for large amounts
of training data for the problem domain. Wang et al. used Hinton’s DBN model in a cyclic architecture instead of
the single DBN model that was earlier presented by Hinton
41
. However, while they used MTS data in their exper-
iments, their target was a classication problem and not a forecasting problem, such as our target in this paper. In
addition, their model architecture is completely dierent from ours as presented here.
Overview of the reference models. Furthermore, given the complexity of MTS data that are described
in the previous section, few works have been introduced to solve forecasting applications using MTS data-
sets. Indeed, specic methodologies are required for treatment rather than traditional parametric methods,
auto-regressive methods, and Gaussian models
2,42
. In the following, we briey review previous contributions in
the literature that are used to solve the same forecasting problems that are studied in this paper.
Wu et al. presented a hybrid prediction model that combines a natural computation algorithm, namely, the
articial immune system (AIS) with regression trees (RT)
43
. e cells in AIS represent the basic constituent ele-
ments of the model and the RT forecasting sub-models are embedded in the AIS model to form the cells’ pool.
ey applied their hybrid model to solve the bike sharing system problem, which we also solve in this paper.
Huang et al. developed a deep neural network model that integrates the CNN and LSTM architectures to solve
the PM2.5 mass concentrations forecasting problem
44
. In this simple combination, the CNN is used for feature
extraction and the LSTM is used to analyse the features extracted by the CNN, and then predict the PM2.5
concentration of the next observation. e authors added a batch normalization step to improve the forecasting
accuracy. e proposed hybrid approach outperformed the single CNN and LSTM models.
Qi et al. proposed a hybrid model that integrates graph convolutional networks (GCN) and long short-term
memory (LSTM) networks to model and forecast the spatio-temporal variation of the PM2.5 mass concentra-
tions forecasting problem
45
. e role of the GCN is to extract the spatial dependency between dierent stations,
whereas the role of the LSTM is to capture the temporal dependency among observations at dierent times. In a
subsequent section of our paper, we also treat the PM2.5 concentrations problem. Cheng et al. developed three
hybrid models combine statistical learning approaches with machine learning approaches to forecast the PM2.5
mass concentrations problem
46
. ese models are: wavelet-ANN, wavelet-ARIMA, and wavelet-SVM. eir
experimental results showed that the proposed hybrid models can signicantly improve the prediction accuracy
better than their single counterpart models. In addition, the wavelet-ARIMA model has the highest accuracy
compared to the other two hybrid models.
Last, Zhao et al. proposed a hybrid method that combines ve modules to solve the PM2.5 mass concentra-
tions forecasting problem
47
. ese ve modules are the data preprocessing module, feature selection module,
optimization module, forecasting module and evaluation module. ey used the complete ensemble empirical
mode decomposition with adaptive noise and variational mode decomposition (CEEMDAN-VMD) to decom-
pose, reconstruct, identify and select the main features of PM2.5 observations through the data preprocessing
module. en, they used the auto-correlation function (ACF) to extract the variables that have relatively large
correlation with the predictor, thereby selecting the input variables according to the order of correlation coe-
cients. Next, the least squares support vector machine (LSSVM) is used to predict the future points of concen-
trations. Finally, the parameters of the LSSVM are optimized using the whale optimization algorithm (WOA).
It is clear that most of the aforementioned state-of-the-art methods in the literature that are used to solve
the same MTS forecasting problems are hybrid methods that combine more than a model, in contrast to the
model proposed in this paper, which is a standalone model. In summary, undoubtedly, the layer-wise architecture
that was earlier presented by Hinton has a concrete theoretical foundation as well as an empirical assessment.