Empirical Analysis of Collaborative Filtering-based
Recommenders in Temporally Evolving Systems
Xiao-Yu Shi, Xin Luo, Ming-Sheng Shang, Xin-Yi Cai
Chongqing Key Laboratory of Big Data and Intelligent Computing
Chongqing Institute of Green and Intelligent Technology
Chinese Academy of Sciences
Chongqing, China
{xiaoyushi, luoxin21, msshang, caixinyi}@cigit.ac.cn
Abstract—Recommender systems benefit people’s daily
lives at every moment. While considerable attentions have been
drawn by performance in one-step recommendation and static
user-item network, recommenders’ performance on temporally
evolving networks remains unclear. To address this issue, this
paper firstly adopts a bipartite network to describe the online
commercial system. We then propose a network evolution
method to simulate the mutual feedback between
recommender system and its users’ decisions in the evolving
network with time. To investigate the long-term performance
of three state-of-the-art CF-based recommenders, i.e., the user-
based collaborative filtering (UCF), item-based collaborative
filtering (ICF) and latent factor-based model (LFM), this
online network is evolving with time driven by each tested
recommender. Besides using root mean squared error (RMSE)
to evaluate prediction accuracy of recommender, we also
calculate the intra-similarity and popularity to study the
performance of recommendation, as well as Gini coefficient to
evaluate the health of online network. Experiments on two real
datasets, we find that during the temporal evolving process
LFM’s accuracy loss is less than that of UCF and ICF, besides
LFM enjoys a high accuracy in one-step recommendation.
Moreover, although LFM proves to be highly accurate and
stable during the temporal evolving network, ICF shows a
better performance than LFM in terms of recommendation
diversity, and it simultaneously benefits the health of online
system. Hence, these results provide insights for the design of a
next generation of recommender systems, which would trade-
offs between short- and long-term performances.
Keywords—Recommender Systems; Online Systems;
Collaborative Filtering; Evolving Network; Temporal Dynamics;
Recommendation Performance
I.
I
NTRODUCTION
With the exponential growth of the World Wide Web,
people confront with the problem of information overload:
available information is too much to fit out the key part [1-2].
In such context, various information filtering tools have been
invented. Among them, recommender systems, which
connect people with their potential favorites by efficiently
analyzing various historical data, prove to be highly effective
[3]. Hence, most popular web sites, e.g., Amazon, Netflix
and YouTube, all adopt recommender systems to improve
users' experience and satisfaction, as well as benefit their
profits [4].
In the last two decades, different kinds of recommenders
have been proposed based on various ideas and concepts.
Among them, collaborative filtering (CF) is a widely-used
category of recommender [5-6]. According to the recent
progress in CF techniques, CF algorithms are dominated by
two different types of models: the Neighborhood Based
Model (NBM) and the Latent Factor Model (LFM).
Specifically, NBM constructs the relationships between users
or items to build the neighborhoods of corresponding entities,
and makes predictions based on the known ratings by the
active user's neighbors, or those on the active item's
neighbors. This can be done in two ways known as user-
oriented (U-NBM) or item-oriented (I-NBM)
recommendation [6]. On the other hand, LFM based
recommenders work by mapping both items and users into
the same latent factor space, and then training the feature
factors to fit the known ratings; the predictions for unknown
ratings will depend on the inner products of the
corresponding user-item feature-vector pairs and some other
factors [7-8]. During the Netflix prize, matrix factorization
(MF)-based techniques for latent factor (LF) analysis on
sparse matrices arise and become the most successful
approach for LFM [9]. The existing literature on this topic
embodies a variety of approaches, including the SVD++
model [10], the Regularized Matrix Factorization (RMF)
[11], probabilistic MF model [12] and so on. Moreover, for
further improving the computational efficiency of LFM-
based recommenders, Luo et al. [13] proposed the alternating
direction method (ADM)-based nonnegative latent factor
(ANLF) model to address these issues. LFM based
recommenders have proved to be highly accurate and
scalable. Nevertheless, NBM based recommenders are
explainable and easy to implement. Thus, these two kinds of
models can be employed in different occasions depending on
detailed requirements.
Considering the design of optimal CF recommender,
most prior works focus on evaluating the one-step
recommendation performance of a recommender system on
the static dataset, which assumes that the rating matrix is
fixed without incremental data. For real-world online
applications, this attitude is lack of practicability since the
Corresponding Author: X. Luo (luoxin21@cigit.ac.cn).
This research is supported in part by the Pioneer Hundred Talents
Program of Chinese Academy of Sciences, in part by the National Natural
Science Foundation of China under Grant No. 91646114, No. 61602434,
o. 61370150 and 61402198, in part by the Youth Innovation Promotion
Association of Chinese Academy of Sciences, No. 2017393, and in part by
the Young Scientist Foundation of Chongqing under Grant No.
cstc2014kjrc-qnrc40005, in part by the Chongqing Research Program of
Basic Research and Frontier Technology under Grant cstc2015jcyjB0244,
in part by the Fundamental Research Funds for the Central Universities
under Grant No. 106112016CDJXY180005 and No. CDJZR12180012.
978-1-5090-4429-0/17/$31.00 ©2017 IEEE.