探索Apache Mahout：机器学习实战指南

需积分: 10 109 浏览量更新于2024-07-29 收藏 5.26MB PDF 举报

"mahout\MahoutinAction英文版，一本关于Apache Mahout的详细指南" Apache Mahout是一个基于Apache Hadoop的数据挖掘库，专注于提供机器学习算法的实现，包括推荐系统、聚类和分类。这本书《Mahout in Action》深入介绍了这三个主要领域，并提供了实践指导。在"Meet Apache Mahout"章节中，作者向读者介绍了Mahout的基本概念，阐述了它在实际应用中的价值。Mahout旨在使大规模数据挖掘变得简单，通过利用Hadoop的分布式计算能力，可以处理海量数据集。该章节可能涵盖了如何安装和配置Mahout，以便读者能够开始自己的数据挖掘项目。推荐系统部分（Part 1 Recommendations）从介绍推荐系统的概念开始，讲解如何在各种场景下（如电影推荐、商品推荐等）构建推荐系统。这部分将涉及数据的表示方法，如用户-项目交互矩阵，以及如何使用协同过滤等算法生成个性化推荐。此外，还会讨论如何将推荐系统部署到生产环境以及如何进行分布式推荐计算，以提高性能和处理能力。聚类部分（Part 2 Clustering）涵盖了聚类算法的基础知识，如K-means、层次聚类等。这部分会解释如何对数据进行有效的表示，以便于聚类算法处理，同时介绍评估聚类质量的方法，如轮廓系数。还将探讨如何将聚类模型应用于实际问题，以及在生产环境中实施聚类算法的策略。分类部分（Part 3 Classification）介绍了监督学习的概念，包括训练分类器（如朴素贝叶斯、决策树等）、评估和调整分类器的性能，以及如何部署分类器到生产环境。通过案例研究，例如"ShopItToMe"，读者可以了解分类技术在实际业务中的应用。附录部分提供了JVM调优的建议，这对于优化Mahout在大型数据集上的运行至关重要。此外，还包含了对Mahout中使用的数学概念的简要概述，以及相关资源列表，帮助读者进一步扩展学习。《Mahout in Action》这本书对于想要掌握Apache Mahout以及机器学习在大数据领域应用的读者来说，是一份宝贵的资源。通过阅读和实践书中的例子，读者不仅可以理解这些算法的工作原理，还能学会如何在实际项目中有效地运用它们。

Last saved: 1/19/2011 Owen and Anil/Mahout in Action

For brevity, through several more chapters of examples that follow, code listings will omit the imports,

class declaration, and method declaration, and instead repeat only the program statements themselves.

To help visualize the relationship between these basic components, see figure 2.2. Not all Mahout-based

recommenders will look like this -- some will employ different components with different relationships. But

this gives a sense of what’s going on in the example.

Figure 2.2 Simplified illustration of component interaction in a Mahout user-based recommender

Before discussing each of these components in more detail in the next two chapters, we can

summarize the role of each component now. A

DataModel

implementation stores and provides access to

all the preference, user and item data needed in the computation. A

UserSimiliarity

implementation

provides some notion of how similar two users are; this could be based on one of many possible metrics

or calculations. A

UserNeighborhood

implementation defines a notion of a group of users that are most

similar to a given user. Finally, a

Recommender

implementation pulls all these components together to

recommend items to users, and related functionality.

2.2.3 Analyzing the output

Compile and run this using your favorite IDE. The output of running the program in your terminal or IDE

should be: RecommendedItem[item:104, value:4.257081]

The request asked for one top recommendation, and got one. The recommender engine recommended

book 104 to user 1. Further, it says that the recommender engine did so because it estimated user 1’s

preference for book 104 to be about 4.3, and that was the highest among all the items eligible for

recommendations.

That’s not bad. 107 did not appear, which was also recommendable, but only associated to a user with

different tastes. It picked 104 over 106, and this makes sense after noting that 104 is a bit more highly

rated overall. Further, the output contained a reasonable estimate of how much user 1 likes item 104 –

something between the 4.0 and 4.5 that users 4 and 5 expressed.

The right answer isn't obvious from looking at the data, but the recommender engine made some

decent sense of it and returned a defensible answer. If you got a pleasant tingle out of seeing this simple

program give a useful and non-obvious result from a small pile of data, then the world of machine learning

is for you!

For clear, small data sets, producing recommendations is as trivial as it appears above. In real life,

data sets are huge, and they are noisy. For example, imagine a popular news site recommending news

articles to readers. Preferences are inferred from article clicks. But, many of these “preferences” may be

bogus – maybe a reader clicked an article but didn't like it, or, had clicked the wrong story. Perhaps many

of the clicks occurred while not logged in, so can’t be associated to a user. And, imagine the size of the

data set – perhaps billions of clicks in a month.

Producing the right recommendations from this data and producing them quickly are not trivial. Later

we will present the tools Mahout provides to attack a range of such problems by way of case studies. They

will show how standard approaches can produce poor recommendations or take a great deal of memory

and CPU time, and, how to configure and customize Mahout to improve performance.

Last saved: 1/19/2011 Owen and Anil/Mahout in Action

2.3 Evaluating a recommender

A recommender engines is a tool, a means to answer the question, “what are the best recommendations

for a user?” Before investigating the answers, it’s best to investigate the question. What exactly is a good

recommendation? And how does one know when a recommender is producing them? The remainder of

this chapter pauses to explore evaluation of a recommender, because this is a tool that will be useful

when looking at specific recommender systems.

The best possible recommender would be a sort of psychic that could somehow know, before you do,

exactly how much you would like every possible item that you've not yet seen or expressed any

preference for. A recommender that could predict all your preferences exactly would merely present all

other items ranked by your future preference and be done. These would be the best possible

recommendations.

And indeed most recommender engines operate by trying to do just this, estimating ratings for some

or all other items. So, one way of evaluating a recommender's recommendations is to evaluate the quality

of its estimated preference values – that is, evaluating how closely the estimated preferences match the

actual preferences.

2.3.1 Training data and scoring

Those “actual preferences” don't exist though. Nobody knows for sure how you'll like some new item in

the future (including you). This can be simulated to a recommender engine by setting aside a small part of

the real data set as test data. These test preferences are not present in the training data fed into a

recommender engine under evaluation -- which is all data except the test data. Instead, the recommender

is asked to estimate preference for the missing test data, and estimates are compared to the actual

values.

From there, it is fairly simple to produce a kind of “score” for the recommender. For example it’s

possible to compute the average difference between estimate and actual preference. With a score of this

type, lower is better, because that would mean the estimates differed from the actual preference values

by less. 0.0 would mean perfect estimation -- no difference at all between estimates and actual values.

Sometimes the root-mean-square of the differences is used: this is the square root of the average of

the squares of the differences between actual and estimated preference values. Again, lower is better.

Table 2.1 An illustration of the average difference, and root mean square calculation

Item 1 Item 2 Item 3

Actual

3.0 5.0 4.0

Estimate

3.5 2.0 5.0

Difference

0.5 3.0 1.0

Average

Difference

= (0.5 + 3.0 + 1.0) / 3 = 1.5

Root Mean

Square

=√((0.5

+ 3.0

+ 1.0

) / 3) = 1.8484

Above, the table shows the difference between a set of actual and estimated preferences, and how

they are translated into scores. Root-mean-square more heavily penalizes estimates that are way off, as

with item 2 here, and that is considered desirable by some. For example, an estimate that’s off by 2 whole

stars is probably more than twice as “bad” as one off by just 1 star. Because the simple average of

differences is perhaps more intuitive and easy to understand, upcoming examples will use it.

2.3.2 Running RecommenderEvaluator

Let's revisit the example code and instead evaluate the simple recommender, on this simple data set.

Last saved: 1/19/2011 Owen and Anil/Mahout in Action

Listing 2.3 Configuring and running an evaluation of a Recommender

RandomUtils.useTestSeed(); A

DataModel model = new FileDataModel(new File("intro.csv"));

RecommenderEvaluator evaluator =

new AverageAbsoluteDifferenceRecommenderEvaluator();

RecommenderBuilder builder = new RecommenderBuilder() { B

@Override

public Recommender buildRecommender(DataModel model)

throws TasteException {

UserSimilarity similarity = new PearsonCorrelationSimilarity(model);

UserNeighborhood neighborhood =

new NearestNUserNeighborhood(2, similarity, model);

return

new GenericUserBasedRecommender(model, neighborhood, similarity);

}

};

double score = evaluator.evaluate(

builder, null, model, 0.7, 1.0); C

System.out.println(score);

A Used only in examples for repeatable result

B Builds the same Recommender as above

C Use 70% of data to train; test with other 30%

Most of the action happens in

evaluate()

. Inside, the

RecommenderEvaluator

handles splitting

the data into a training and test set, builds a new training

DataModel

and

Recommender

to test, and

compares its estimated preferences to the actual test data.

Note that there is no

Recommender

passed to this method. This is because, inside, the method will

need to build a

Recommender

around a newly created training

DataModel

. So the caller must provide an

object that can build a

Recommender

from a

DataModel

– a

RecommenderBuilder

. Here, it builds the

same implementation that was tried earlier in this chapter.

2.3.3 Assessing the result

This program prints the result of the evaluation: a score indicating how well the

Recommender

performed. In this case you should simply see:

1.0

. Even though a lot of randomness is used inside the

evaluator to choose test data, the result should be consistent because of the call to

RandomUtils.useTestSeed()

, which forces the same random choices each time. This is only used in

such examples, and unit tests, to guarantee repeatable results. Don’t use it in your real code.

What this value means depends on the implementation used – here,

AverageAbsoluteDifferenceRecommenderEvaluator

. A result of 1.0 from this implementation

means that, on average, the recommender estimates a preference that deviates from the actual

preference by 1.0.

A value of 1.0 is not great, on a scale of 1 to 5, but there is so little data here to begin with. Your

results may differ as the data set is split randomly, and hence the training and test set may differ with

each run.

This technique can be applied to any

Recommender

and

DataModel

. To use root-mean-square

scoring, replace

AverageAbsoluteDifferenceRecommenderEvaluator

with the implementation

RMSRecommenderEvaluator

Also, the

null

parameter to

evaluate()

could instead be an instance of

DataModelBuilder

which can be used to control how the training

DataModel

is created from training data. Normally the

default is fine; it may not be if you are using a specialized implementation of

DataModel

in your

deployment. A

DataModelBuilder

is how you would inject it into the evaluation process.

The

1.0

parameter at the end controls how much of the overall input data is used. Here it means

“100%.” This can be used to produce a quicker, if less accurate, evaluation by using only a little of a

potentially huge data set. For example,

0.1

would mean 10% of the data is used and 90% is ignored.

This is quite useful when rapidly testing small changes to a

Recommender

Last saved: 1/19/2011 Owen and Anil/Mahout in Action

2.4 Evaluating precision and recall

We could also take a broader view of the recommender problem: it’s not strictly necessary to estimate

preference values in order to produce recommendations. It’s not always essential to present estimated

preference values to users. In many cases, just an ordered list of recommendations, from best to worst, is

sufficient. In fact, in some cases the exact ordering of the list doesn’t matter much – a set of a few good

recommendations is fine.

Taking this more general view, we could also apply classic information retrieval metrics to evaluate

recommenders: precision and recall. These terms are typically applied to things like search engines, which

return some set of best results for a query out of many possible results.

A search engine should not return irrelevant results in the top results, although it should strive to

return as many relevant results as possible. “Precision” is the proportion of top results that are relevant,

for some definition of relevant. “Precision at 10” would be this proportion judged from the top 10 results.

“Recall” is the proportion of all relevant results included in the top results. See figure 2.3 for a

visualization of these ideas.

Figure 2.3 An illustration of precision and recall in the context of search results

These terms can easily be adapted to recommenders: precision is the proportion of top

recommendations that are good recommendations, and recall is the proportion of good recommendations

that appear in top recommendations. The next section will define “good”.

2.4.1 Running RecommenderIRStatsEvaluator

Again, Mahout provides a fairly simple way to compute these values for a

Recommender

Listing 2.4 Configuring and running a precision and recall evaluation

RandomUtils.useTestSeed();

DataModel model = new FileDataModel(new File("intro.csv"));

RecommenderIRStatsEvaluator evaluator =

new GenericRecommenderIRStatsEvaluator();

RecommenderBuilder recommenderBuilder = new RecommenderBuilder() {

@Override

public Recommender buildRecommender(DataModel model)

throws TasteException {

UserSimilarity similarity = new PearsonCorrelationSimilarity(model);

UserNeighborhood neighborhood =

new NearestNUserNeighborhood(2, similarity, model);

return

new GenericUserBasedRecommender(model, neighborhood, similarity);

}

};

IRStatistics stats = evaluator.evaluate(

recommenderBuilder, null, model, null, 2,

GenericRecommenderIRStatsEvaluator.CHOOSE_THRESHOLD,

1.0); A

Last saved: 1/19/2011 Owen and Anil/Mahout in Action

System.out.println(stats.getPrecision());

System.out.println(stats.getRecall());

A Evaluate precision and recall at 2

Without the call to

RandomUtils.useTestSeed()

, the result you see would vary significantly due to

random selection of training data and test data, and because the data set is so small here. But with the

call, the result ought to be:

0.75

1.0

Precision at 2 is 0.75; on average about three-quarters of recommendations were “good.” Recall at 2 is

1.0; all good recommendations are among those recommended.

But what exactly is a “good” recommendation here? The framework was asked to decide. It didn’t

receive a definition. Intuitively, the most highly preferred items in the test set are the good

recommendations, and the rest aren’t.

Listing 2.5 User 5’s preference in test data set

5,101,4.0

5,102,3.0

5,103,2.0

5,104,4.0

5,105,3.5

5,106,4.0

Look at user 5 in this simple data set again. Let’s imagine the preferences for items 101, 102 and 103

were withheld as test data. The preference values for these are 4.0, 3.0 and 2.0. With these values

missing from the training data, a recommender engine ought to recommend 101 before 102, and 102

before 103, because this is the order in which user 5 prefers these items. But would it be a good idea to

recommend 103? It’s last on the list; user 5 doesn’t seem to like it much. Book 102 is just average. Book

101 looks reasonable as its preference value is well above average. Maybe 101 is a good

recommendation; 102 and 103 are valid, but not good recommendations.

And this is the thinking that the

RecommenderEvaluator

employs. When not given an explicit

threshold that divides good recommendations from bad, the framework will pick a threshold, per user,

that is equal to the user's average preference value µ plus one standard deviation σ:

threshold = µ +

If you’ve forgotten your statistics, don’t worry. This takes items whose preference value is not merely

a little more than average (µ), but above average by a significant amount (σ). In practice this means that

about the 16% of items that are most highly preferred are considered “good” recommendations to make

back to the user. The other arguments to this method are similar to those discussed before and are more

fully documented in the project javadoc.

2.4.2 Problems with precision and recall

The usefulness of precision and recall tests in the context of recommenders depends entirely on how well

a “good” recommendation can be defined. Above, the threshold was given, or defined by the framework. A

poor choice will hurt the usefulness of the resulting score.

There’s a subtler problem with these tests, though. Here, they necessarily pick the set of good

recommendations from among items for which the user has already expressed some preference. But the

best recommendations are of course not necessarily among those the user already knows about!

Imagine running such a test for a user who would absolutely love the little-known French cult film, “My

Brother The Armoire”. Let’s say it’s objectively a great recommendation for this user. But, the user has

never heard of this movie. If a recommender actually returned this film when recommending movies, it

剩余286页未读，继续阅读

wff19901226

粉丝: 0
资源: 22

探索Apache Mahout：机器学习实战指南

mahout-core-0.9.jar+mahout-core-0.8.jar+mahout-core-0.1.jar

MahoutInAction中文版啦word版

Mahout In Action英文完整版

win 10下载并配置Mahout

mahout可以用python吗

apache mahout协同过滤

【Apache 之Mahout介绍】

mahout实现KNN的代码

在python中如何使用Mahout库

基于Mahout实现K-Means聚类

最新资源