Mahout实战指南：推荐与聚类应用详解

需积分: 10 173 浏览量更新于2024-07-29 收藏 2.41MB PDF 举报

《Mahout in Action》是一本深入讲解Apache Mahout开源机器学习库的实战指南。本书由Manning Publications出版，适用于希望在推荐系统和数据挖掘领域应用机器学习技术的读者。试读版包含了以下几个关键章节： 1. ** Meet Mahout**：首先介绍Mahout的基本概念和背景，包括它在大数据处理中的地位，以及为何在推荐系统和聚类分析等领域具有重要作用。 2. ** Recommenders **：这部分着重于推荐算法，如协同过滤（Collaborative Filtering）的基础原理，以及如何利用Mahout实现个性化推荐，包括基于用户和物品的推荐方法。 3. ** Representing Data **：章节探讨了数据预处理和表示的重要性，如如何将用户行为、物品特征转化为可供模型使用的向量形式。 4. ** Making Recommendations **：介绍了如何利用计算模型（如矩阵分解或基于内容的推荐）生成推荐列表，并可能涉及实时和离线推荐的区别。 5. ** Taking recommenders to production **：讨论了将推荐系统部署到实际环境中的策略，包括性能优化、扩展性和用户体验的考虑。 6. ** Distributing Recommendation Computations **：这一部分关注分布式计算，说明如何在集群环境下并行处理大规模数据，提升推荐性能。 7. ** Introduction to Clustering **：接着转向聚类分析，讲述了其在数据挖掘中的作用，以及Mahout如何支持各种聚类算法。 8. ** Representing Data for Clustering **：同样强调数据的表示和转换对于聚类算法的有效性。 9. ** Clustering algorithms in Mahout **：详细介绍了Mahout支持的几种聚类算法，如K-Means、DBSCAN等，并演示如何在实际项目中应用。 10. ** Evaluating cluster quality **：讨论评估聚类效果的方法，如内部评估指标和外部评估标准。 11. ** Taking clustering to production **：总结将聚类技术引入生产环境的步骤和注意事项。 12. ** Real-world applications of clustering **：展示了聚类在实际场景中的应用案例，如市场分割、用户行为理解等。 13. ** Introduction to classification **：转向分类问题，讲解基础的分类算法如朴素贝叶斯，以及在Mahout中的实现。 14. ** Power of the Naive Classifier **：强调朴素贝叶斯分类器的简单性和有效性，尤其是在文本分类中的应用。 15. ** Multi-class classification **：探讨多类别分类问题，包括处理不平衡数据集和选择合适的分类算法。 16. ** Classifier evaluation **：讨论评估分类模型性能的关键指标，如准确率、召回率和F1分数。 17. ** Tuning your classifier for greater accuracy and performance **：提供调优技巧，以提升模型的预测精度和效率。这本书不仅涵盖了Mahout的基础知识，还提供了实践经验，适合对机器学习感兴趣并希望应用于推荐系统和数据分析领域的专业人士使用。通过阅读试读版，读者可以了解到如何在实际项目中利用Mahout解决复杂的数据处理问题。

©Manning Publications Co. Please post comments or corrections to the Author Online forum:

http://www.manning-sandbox.com/forum.jspa?forumID=623

Figure 2.1 Relationships between users 1 to 5 and items 101 to 107. Dashed lines represent associations that seem

negative -- the user does not seem to like the item much, but expresses a relationship to the item.

2.2.2 Creating a Recommender

So what book might we recommend to user 1? Not 101, 102 or 103 – he already knows about these

books, apparently, and recommendation is about discovering new things. Intuition suggests that

because users 4 and 5 seem similar to 1, we should recommend something that user 4 or user 5 likes.

That leaves 104, 105 and 106 as possible recommendations. On the whole, 104 seems to be the most

liked of these possibilities, judging by the preference values of 4.5 and 4.0 for item 104. Now, run the

following code:

Listing 2.2 A simple user-based recommender program with Mahout

package mia.recommender.ch02;

import org.apache.mahout.cf.taste.impl.model.file.*;

import org.apache.mahout.cf.taste.impl.neighborhood.*;

import org.apache.mahout.cf.taste.impl.recommender.*;

import org.apache.mahout.cf.taste.impl.similarity.*;

import org.apache.mahout.cf.taste.model.*;

import org.apache.mahout.cf.taste.neighborhood.*;

import org.apache.mahout.cf.taste.recommender.*;

import org.apache.mahout.cf.taste.similarity.*;

import java.io.*;

import java.util.*;

class RecommenderIntro {

public static void main(String[] args) throws Exception {

DataModel model = new FileDataModel(new File("intro.csv")); A

UserSimilarity similarity = new PearsonCorrelationSimilarity(model);

UserNeighborhood neighborhood =

new NearestNUserNeighborhood(2, similarity, model);

Recommender recommender = new GenericUserBasedRecommender(

model, neighborhood, similarity); B

List<RecommendedItem> recommendations =

recommender.recommend(1, 1); C

for (RecommendedItem recommendation : recommendations) {

System.out.println(recommendation);

}

A Load the data file

B Create the recommender engine

C For user 1, recommend 1 item

Licensed to nancy chen <amigo4u2009@gmail.com>

©Manning Publications Co. Please post comments or corrections to the Author Online forum:

http://www.manning-sandbox.com/forum.jspa?forumID=623

For brevity, through several more chapters of examples that follow, we will omit the imports, class

declaration, and method declaration, and instead repeat only the program statements themselves. To

help visualize the relationship between these basic components, see figure 2.2. Not all Mahout-based

recommenders will look like this -- some will employ different components with different relationships.

But this gives a sense of what’s going on in our example.

Figure 2.2 Simplified illustration of component interaction in a Mahout user-based recommender

While we will discuss each of these components in much more detail in the next two chapters, we

can summarize the role of each component now. A

DataModel implementation stores and provides

access to all the preference, user and item data needed in the computation. A

UserSimiliarity

implementation provides some notion of how similar two users are; this could be based on one of many

possible metrics or calculations. A

UserNeighborhood implementation defines a notion of a group of

users that are most similar to a given user. Finally, a

Recommender implementation pulls all these

components together to recommend items to users, and related functionality.

2.2.3 Analyzing the output

Compile and run this using your favorite IDE. The output of running the program in your terminal or IDE

should be: RecommendedItem[item:104, value:4.257081]

We asked for one top recommendation, and got one. The recommender engine recommended book

104 to user 1. Further, it says that the recommender engine did so because it estimated user 1’s

preference for book 104 to be about 4.3, and that was the highest among all the items eligible for

recommendations.

That’s not bad. We didn't get 107, which was also recommendable, but only associated to a user

with different tastes. We picked 104 over 106, and this makes sense when you note that 104 is a bit

more highly rated overall. Further, we got a reasonable estimate of how much user 1 likes item 104 –

something between the 4.0 and 4.5 that users 4 and 5 expressed.

The right answer isn't obvious from looking at the data, but the recommender engine made some

decent sense of it and returned a defensible answer. If you got a pleasant tingle out of seeing this

simple program give a useful and non-obvious result from a small pile of data, then the world of

machine learning is for you!

For clear, small data sets, producing recommendations is as trivial as it appears above. In real life,

data sets are huge, and they are noisy. For example, imagine a popular news site recommending news

Licensed to nancy chen <amigo4u2009@gmail.com>

©Manning Publications Co. Please post comments or corrections to the Author Online forum:

http://www.manning-sandbox.com/forum.jspa?forumID=623

articles to readers. Preferences are inferred from article clicks. But, many of these “preferences” may be

bogus – maybe a reader clicked an article but didn't like it, or, had clicked the wrong story. Perhaps

many of the clicks occurred while not logged in, so can’t be associated to a user. And, imagine the size

of the data set – perhaps billions of clicks in a month.

Producing the right recommendations from this data and producing them quickly are not trivial. Later

we will present the tools Mahout provides to attack a range of such problems by way of case studies.

They will show how standard approaches can produce poor recommendations or take a great deal of

memory and CPU time, and, how to configure and customize Mahout to improve performance.

2.3 Evaluating a Recommender

A recommender engines is a tool, a means to answer the question, “what are the best recommendations

for a user?” Before investigating the answers, we should investigate the question. What exactly is a

good recommendation? And how will we know when a recommender is producing them? The remainder

of this chapter pauses to explore evaluation of a recommender, because this is a tool that will be useful

when we begin looking at specific recommender systems.

The best possible recommender would be a sort of psychic that could somehow know, before you do,

exactly how much you would like every possible item that you've not yet seen or expressed any

preference for. A recommender that could predict all your preferences exactly would merely present all

other items ranked by your future preference and be done. These would be the best possible

recommendations.

And indeed most recommender engines operate by trying to do just this, estimating ratings for some

or all other items. So, one way of evaluating a recommender's recommendations is to evaluate the

quality of its estimated preference values – that is, evaluating how closely the estimated preferences

match the actual preferences.

2.3.1 Training data and scoring

Those “actual preferences” don't exist though. Nobody knows for sure how you'll like some new item in

the future (including you). This can be simulated to a recommender engine by setting aside a small part

of the real data set as test data. These test preferences are not present in the training data fed into a

recommender engine under evaluation -- which is all data except the test data. Instead, the

recommender is asked to estimate preference for the missing test data, and estimates are compared to

the actual values.

From there, it is fairly simple to produce a kind of “score” for the recommender. For example we

could compute the average difference between estimate and actual preference. With a score of this

type, lower is better, because that would mean the estimates differed from the actual preference values

by less. 0.0 would mean perfect estimation -- no difference at all between estimates and actual values.

Sometimes the root-mean-square of the differences is used: this is the square root of the average of

the squares of the differences between actual and estimated preference values. Again, lower is better.

Licensed to nancy chen <amigo4u2009@gmail.com>

©Manning Publications Co. Please post comments or corrections to the Author Online forum:

http://www.manning-sandbox.com/forum.jspa?forumID=623

Item 1 Item 2 Item 3

Actual

3.0 5.0 4.0

Estimate

3.5 2.0 5.0

Difference

0.5 3.0 1.0

Average

Difference

= (0.5 + 3.0 + 1.0) / 3 = 1.5

Root Mean

Square

=√((0.52 + 3.02 + 1.02) / 3) = 1.8484

Table 2.1 An illustration of the average difference, and root mean square calculation

Above, the table shows the difference between a set of actual and estimated preferences, and how

they are translated into scores. Root-mean-square more heavily penalizes estimates that are way off, as

with item 2 here, and that is considered desirable by some. For example, an estimate that’s off by 2

whole stars is probably more than twice as “bad” as one off by just 1 star. Because the simple average

of differences is perhaps more intuitive and easy to understand, we’ll use it in upcoming examples.

2.3.2 Running RecommenderEvaluator

Let's revisit the example code and instead evaluate the simple recommender we created, on our simple

data set:

Listing 2.3 Configuring and running an evaluation of a Recommender

RandomUtils.useTestSeed(); A

DataModel model = new FileDataModel(new File("intro.csv"));

RecommenderEvaluator evaluator =

new AverageAbsoluteDifferenceRecommenderEvaluator();

RecommenderBuilder builder = new RecommenderBuilder() { B

@Override

public Recommender buildRecommender(DataModel model)

throws TasteException {

UserSimilarity similarity = new PearsonCorrelationSimilarity(model);

UserNeighborhood neighborhood =

new NearestNUserNeighborhood(2, similarity, model);

return

new GenericUserBasedRecommender(model, neighborhood, similarity);

}

};

double score = evaluator.evaluate(

builder, null, model, 0.7, 1.0); C

System.out.println(score);

A Used only in examples for repeatable result

B Builds the same Recommender as above

C Use 70% of data to train; test with other 30%

Licensed to nancy chen <amigo4u2009@gmail.com>

http://www.manning-sandbox.com/forum.jspa?forumID=623

Most of the action happens in evaluate(). Inside, the RecommenderEvaluator handles splitting

the data into a training and test set, builds a new training

DataModel and Recommender to test, and

compares its estimated preferences to the actual test data.

Note that we don’t pass a

Recommender to this method. This is because, inside, the method will

need to build a

Recommender around a newly created training DataModel. So we must provide an

object that can build a

Recommender from a DataModel – a RecommenderBuilder. Here, it builds

the same implementation that we tried in the first chapter.

2.3.3 Assessing the result

This program prints the result of the evaluation: a score indicating how well the Recommender

performed. In this case you should simply see:

1.0. Even though a lot of randomness is used inside the

evaluator to choose test data, the result should be consistent because of the call to

RandomUtils.useTestSeed(), which forces the same random choices each time. This is only used in

such examples, and unit tests, to guarantee repeatable results. Don’t use it in your real code.

What this value means depends on the implementation we used – here,

AverageAbsoluteDifferenceRecommenderEvaluator. A result of 1.0 from this implementation

means that, on average, the recommender estimates a preference that deviates from the actual

preference by 1.0.

A value of 1.0 is not great, on a scale of 1 to 5, but there is so little data here to begin with. Your

results may differ as the data set is split randomly, and hence the training and test set may differ with

each run.

This technique can be applied to any

Recommender and DataModel. To use root-mean-square

scoring, replace

AverageAbsoluteDifferenceRecommenderEvaluator with the implementation

RMSRecommenderEvaluator.

Also, the

null parameter to evaluate() could instead be an instance of DataModelBuilder,

which can be used to control how the training

DataModel is created from training data. Normally the

default is fine; it may not be if you are using a specialized implementation of

DataModel in your

deployment. A

DataModelBuilder is how you would inject it into the evaluation process.

The

1.0 parameter at the end controls how much of the overall input data is used. Here it means

“100%.” This can be used to produce a quicker, if less accurate, evaluation by using only a little of a

potentially huge data set. For example,

0.1 would mean 10% of the data is used and 90% is ignored.

This is quite useful when rapidly testing small changes to a

Recommender.

2.4 Evaluating precision and recall

We could also take a broader view of the recommender problem: we don't have to estimate preference

values to produce recommendations. It’s not always necessary to present estimated preference values

to users. In many cases, all we want is an ordered list of recommendations, from best to worst. In fact,

in some cases we don't care much about the exact ordering of the list – a set of a few good

recommendations is fine.

Taking this more general view, we could also apply classic information retrieval metrics to evaluate

recommenders: precision and recall. These terms are typically applied to things like search engines,

which return some set of best results for a query out of many possible results.

Licensed to nancy chen <amigo4u2009@gmail.com>

剩余178页未读，继续阅读

wff19901226

粉丝: 0
资源: 23

Mahout实战指南：推荐与聚类应用详解

Mahout In Action英文完整版

Mahout 中文文档

win 10下载并配置Mahout

apache mahout协同过滤

mahout实现KNN的代码

【Apache 之Mahout介绍】

使用Apache Mahout创建在线推荐系统

在python中如何使用Mahout库

基于Mahout实现K-Means聚类

基于mahout协同过滤算法

最新资源