recommendation accuracy on the static dataset, which assumes
that rating matrix is fixed without incremental data. For
real-world online applications and systems, this assumption is
lack of practicability since the mutual feedback between system
and the user decisions are on-going [29]. Thus, the new ratings
happen continuously in recommender systems, and the
real-world sites with recommender system are temporally
evolving systems.
Most importantly, it is one of the most important goals but
not the ultimate goal that making good rating predictions for
unrated items. The original intention of employing
recommender system is that not only mining some useful
information to customer, but also bringing profits to the sites
[30-31]. Hence, the ultimate goal of employing the
recommender system is to broaden the scope of customer
interest, in addition to recommend unrated items with high
predicted rating to user.
In summary, the quality metrics of a healthy recommender
system should include accuracy, novelty, diversity and other
utilities. Furthermore, it is significant to evaluate the quality of
recommender system in a more comprehensive way that the
combination of short- and long-term effects.
Unfortunately, the long-term effects of a CF-based
recommender in the temporally evolving system are rarely
discussed up to date. For example, it is still unclear whether a
well-performed CF-based recommender in a one-step
recommendation can also provide high performance in the
long-term recommendation. Meanwhile, it is not sure that an
accurate-oriented recommender (e.g., an LFM-based one) can
also share the diversity of recommendation and the health of
temporally evolving system. Last but not least, there are several
unknown questions about: whether it will hurt the health of
temporally evolving system by using highly accurate
recommender; and which recommender will lead to this effect;
what extent this effect will be during the temporal evolving
process.
In such context, some works have been proposed to address
the aforementioned issues. In order to deal with customers’
preference for products which are keep drifting over time,
Koren et al. [29] proposed a time-aware factor model that
introducing the temporal dynamics of user rating criteria rating
and item’s popularity. Based on this model, Koenigstein et al.
[17] proposed a rich bias model with consideration of items’
type to improve the recommendation accuracy in Yahoo!
Music dataset.
Since the data of historical user behavior are ever growing in
real-world applications, Luo et al. proposed an incremental
collaborative filtering model to deal with the problem of data
explosion [32]. For the heterogeneity environment, Rosaci et al.
proposed a new agent-based system with consideration of the
effect of the device exploited by the user [33]. Shi et al. [34]
designed a user preference-oriented recommender to deal with
the temporal dynamics in the online system. However, these
works only focus on improving the accuracy of recommender
without considering other important performance indexes, such
as diversity of recommendation and health of system. Different
with above works, we give a comprehensive performance study
of recommender in temporally evolving system.
Moreover, there are several studies along this line. Ekstrand
et al. [35] proposed a novel MovieLens movie recommender
that allows users to choose recommender and studies its
performance in the long-term evolving process. and
Experimental results showed that how customers make use of
this power. In order to evaluate the long-term effects of using
recommender system, Zeng et al. [36] proposed an evolution
model to simulate the interaction between recommender system
and its users, but it not considered the temporal dynamics. Hu et
al. [37] investigated the recommendation accuracy of different
recommenders in evolving networks without considering the
temporal dynamics. Zhao et al. [38] reported the performance
of different recommender on the evolution of user-item
bipartite network. However, the paper focuses mainly on
studying the performance of entity relationships-based
recommender instead of involving numerical
optimization-based ones (e.g., LFM). In addition, the
recommendation with time is iterated for a small number of
rounds, making the long-term effects of co-evolution between
recommender and users decisions hard to detect.
This paper focuses on investigating the long-term
performance of CF-based recommenders in temporally
evolving systems. To do so, we firstly model a recommender
system with an evolving user-item bipartite network, and then
propose a recommendation-based evolution method to simulate
the temporal dynamics between recommender and the
decisions of its users. Meanwhile, the network is evolving
driven by recommender. In such settings, we study the
performance of three well-known CF-based recommenders,
i.e., U-NBM, I-NBM and LFM. For thoroughly evaluating the
performance of involved models, we have adopted four
performance metrics, i.e., Gini coefficient for ecosystem health,
intra-similarity and popularity for recommendation diversity,
and the root mean squared error (MASE) for recommender
accuracy.
To summarize, the main motivation of this work is to explore
the long-term performance of CF-based recommenders in the
temporally evolving system, in order to make them more
practical in real-world application. The main contributions
include:
--We will propose a recommendation-based evolution method
to capture the temporal dynamics between recommender
system and users’ behaviors in the temporally evolving system.
--We will investigate the ecosystem health and diversity of
recommendation as well as recommender accuracy. Through
empirical study, we find that numerical optimization-based
LFM has the lowest accuracy loss during the temporal evolving
process, while entity relationships-based ICF shows a relatively
better performance than LFM in terms of recommendation
diversity and ecosystem health.
--We will conduct experiment on two real datasets for studying
the performance of CF-based recommenders with the proposed
method.
The rest of this paper is organized as follows: Section II
presents the preliminaries. Section III develops the proposed
method. Section IV illustrates the experiments and discusses