Lucene实战：探索搜索引擎核心技术

需积分: 10 135 浏览量更新于2024-08-02 收藏 7.59MB PDF 举报

"Lucene in Action (英文第二版)" 是一本关于Apache Lucene的专著，于2009年出版。这本书深入浅出地介绍了Lucene的使用方法和高级搜索技术，适合对Java搜索库感兴趣的开发者阅读。在本书中，读者可以了解到： 1. **理解Lucene**：Lucene是一个强大的Java全文搜索引擎库，它提供了文本分析、索引和搜索功能，可以帮助开发者快速集成到自己的应用中。Lucene的核心功能包括文本分词、建立倒排索引以及高效的搜索算法。 2. **通用搜索应用架构**：书中会介绍构建搜索应用的一般性架构，包括数据收集、预处理、索引创建、查询解析、结果排序和展示等关键步骤。这对于设计和实现高效、用户友好的搜索系统至关重要。 3. **基本的索引API**：学习如何使用Lucene的API来创建、更新和管理索引。这涵盖了添加文档、删除文档以及优化索引等操作，使开发者能够掌握构建索引的基础知识。 4. **使用搜索API**：通过实际示例，书里将演示如何使用Lucene的搜索API进行查询、过滤和评分。这些API使得开发者能够实现复杂的查询逻辑，如布尔查询、短语查询、模糊查询和范围查询。 5. **高级搜索技术**：书中深入探讨了如TF-IDF、BM25等评分模型，以及如何实现更精确的召回率和准确率。此外，还涉及到了如近似搜索、拼写纠错、相关性排名等高级主题。 6. **扩展搜索**：介绍了如何利用Lucene的可扩展性，开发自定义分析器、过滤器、查询解析器等，以满足特定的搜索需求。 7. **解析常见文档格式**：讨论了Lucene处理PDF、Word、HTML等多种文档格式的能力，帮助开发者索引和搜索非文本内容。 8. **工具和扩展**：涵盖了Lucene社区提供的各种工具和扩展，如索引工具、性能测试工具等，以及如何利用它们提高开发效率和系统性能。 9. **Lucene的移植与变种**：介绍了Solr、Elasticsearch等基于Lucene的搜索服务，以及它们在分布式搜索和集群管理方面的优势。 10. **管理和性能调优**：提供关于索引维护、硬件配置、查询优化等方面的指导，以确保Lucene在生产环境中的高效运行。 11. **案例研究**：通过实际案例，展示Lucene在不同场景下的应用，帮助读者理解如何将所学应用于实际项目。 12. **附录**：包括安装指南、Lucene索引格式的详细解析，以及资源列表和基准测试框架的使用说明。通过阅读《Lucene in Action》第二版，开发者不仅可以掌握Lucene的基本用法，还能深入了解搜索引擎的原理，从而在自己的项目中充分发挥Lucene的功能。

quickly kill a user’s search experience. Be sure the sort order is clearly called out, and defaults to an

appropriate starting point (usually relevance). Be fully transparent: if your search application is doing

something “interesting”, such as expanding the search to include synonyms, using boosts to influence sort

order, or automatically correcting spelling, say so clearly at the top of the search results and make it easy

for the user to turn off.

NOTE

The worst thing that can happen, and it happens quite easily, is to erode the user’s trust in the search

results. Once this happens you may never again have the chance to earn that trust back, and your

users will quietly move on.

Most of all, eat your own dog food: use your own search application extensively yourself. Enjoy

what’s good about it, but aggressively fix the bad things. Almost certainly your search interface should

offer spell correction. Lucene has a sandbox component, spellchecker, covered in section 8.XXX, that you

can use. Likewise, providing dynamic excerpts (sometimes called summaries) with hit highlighting is also

important, and Lucene has a sandbox component, highlighter, covered in section 8.XXX, to handle this.

Lucene does not provide any default search user interface; it’s entirely up to your application to build

this. Once a user interacts with your search interface, she or he submits a search request which first must

be translated into an appropriate

Query

for the search engine.

UILD

UERY

When you finally manage to entice a user to using your search application, she or he issues a search

request, often as the result of an HTML form submitted by a browser to your server. You must then

translate the request into the search engine’s

Query

object. We call this the “Build Query” step.

Query

objects can be very simple or very complex. Lucene provides a powerful package, called

QueryParser

, to process the user’s text into a query object using a common search syntax, described at

http://lucene.apache.org/java/docs/queryparsersyntax.html

. The query may contain Boolean operations,

phrase queries (in double quotes), wildcard terms, etc. If your application has further controls on the

search UI, or further interesting constraints, you must implement logic to translate this into the equivalent

query. For example, if there are entitlement constraints to restrict which set of documents each user is

allowed to search, you’ll need to set up filters on the query.

Many applications will at this point also modify the search query so as to boost or filter for important

things, if the boosting was not done during indexing (see section 1.3.1). Often an ecommerce site will

boost categories of products that are more profitable, or filter out products presently out of stock (so you

don’t see that they are out of stock and then go elsewhere to buy them). Resist the temptation to heavily

boost and filter the search results: users will catch on and lose trust.

Often, for an application, Lucene’s default

QueryParser

is sufficient. Sometimes, you’ll want to use

the output of QueryParser but then add your own logic afterwards to further refine the query object. Still

other times you want to customize the

QueryParser

’s syntax, or customize which

Query

instances it

actually creates, which thanks to Lucene’s open source nature, is straightforward. We discuss customizing

QueryParser

in Section 6.3. Finally, you’re ready to actually execute the search request to retrieve

results.

EARCH

UERY

This is the process of consulting the search index and retrieving the documents matching the

Query

sorted in the requested sort order. This is the very complex inner workings of the search engine, and

Lucene handles all of this magic for you, under a surprisingly simple API. Lucene is also wonderfully

extensible at this point, so if you’d like to customize how results are gathered, filtered, sorted, etc., it’s

straightforward. See chapter 6 “Extending Search” for details.

There are three common models of search:

 Pure boolean model -- Documents either match or do not match the provided query, and no scoring

is done. In this model there are no relevance scores associated with matching documents; a query

simply identifies a subset of the overall corpus as matching the query.

 Vector space model -- Both queries and documents are modeled as vectors in a very high

dimensional space, where each unique term is a dimension. Relevance, or similarity, between a

query and a document is computed by a vector distance measure between these vectors.

 Probabilistic model -- Computes the probability that a document is a good match to a query using a

full probabilistic approach.

Lucene’s approach combines the vector space and pure Boolean models. Lucene returns documents

which you next must render in a very consumable way for your users.

ENDER

ESULTS

Once you have the raw set of documents that match the query, sorted in the right order, you then render

them to the user in an intuitive, consumable manner. Importantly, the user interface should also offer a

clear path for follow-on searches or actions, such as clicking to the next page, refining the search in some

way, finding document similar to one of the matches, etc, so that user never hits a “dead end”. Lucene’s

core doesn’t offer any components to fully render results, but the sandbox contains the highlighter

package, described in section 8.xxx, for producing dynamic summaries and highlighting hits.

We’ve finished reviewing the components of both the indexing and searching paths in a search

application, but we are not yet done. Search applications also often require ongoing administration.

1.3.3 The rest of the search application

Believe it or not, there is in fact still quite a bit more to a typical fully functional search engine, especially

a search engine running on a web site. This includes administration, in order to keep track of the

application’s health, configure the different components, start and stop servers, etc. It also includes

analytics, allowing you to see different views into how your users are using the search feature, thus giving

you the necessary guidance on what’s working and what’s not. Finally, for large search applications,

scaleout in both size of the content you need to search and number of simultaneous search requests, is a

very important feature. Spanning the left side of figure 1.4 is the Administration Interface.

DMINISTRATION

NTERFACE

A modern search engine is a complex piece of software and has numerous controls that need

configuration. If you are using a crawler to discover your content, the administration interface should let

you set the starting URLs, create rules to scope which sites the crawler should visit or which document

types it should load, set how quickly it’s allowed to read documents, etc. Starting and stopping servers,

managing replication (if it’s a high scale search, or, if high availability failover is required), culling search

logs, checking overall system health, creating and restoring from backups, etc., are all examples of what

an administration interface might offer.

Many search applications, such as desktop search, don’t require this component, whereas a full

enterprise search application may have a very complex Administration Interface. Often the interface is

primarily web-based but may also consist of additional command-line tools. On the right side of figure 1.4

is the analytics interface.

NALYTICS

NTERFACE

Spanning the right side is the Analytics Interface, which is often a Web based UI, perhaps running under a

separate server hosting a reporting engine. Analytics is important: you can gain a lot of intelligence about

your users and why they do or do not buy your widgets through your web site, by looking for patterns in

the search logs. Some would say this is the most important reason to deploy a good search engine! If

you are an ecommerce web site, seeing how your users run searches, which searches failed to produce

satisfactory results, which results users clicked on, how often a purchase followed or did not follow a

search, etc, are all incredibly powerful tools to optimize the buying experience of your users. Lucene does

not provide any analytics tools.

If your search application is web-based, Google Analytics is a fast way to create an analytics

interface. If that’s not right, you can also build your own charts based on Google’s visualization API. The

final topic we visit is scaleout.

CALEOUT

One particularly tricky area is scaleout of your search application. The vast majority of search applications

do not have enough content nor simultaneous search traffic to require scaleout beyond a single computer.

Lucene can handle a sizable amount of content on a single modern computer. Still, such applications may

want to run two identical computers to have no single point of failure (no downtime) plus a means of

taking one part of the search out of production to do maintenance, upgrades, etc.

There are two dimensions to scaleout: net amount of content, and net query throughput. If you have

a tremendous amount of content, you must divide it into shards, such that a separate computer searches

each shard. A front end server sends a single incoming query to all shards, and then coalesces the results

into a single result set. If, instead, you have high search throughput during your peaks you’ll have to take

the same index and have multiple computers search it. A front-end load balancer sends each incoming

query to the least loaded back-end computer. If you require both dimensions of scaleout, as a Web scale

search engine will, you combine both of these practices. Both of these require a reliable means of

replicating the search index across computers.

Lucene itself provides no facilities for scaleout. However, both Solr and Nutch and others, which build

additional search engine functionality on top of Lucene, do.

We’ve finished reviewing the components of a modern search application. Now it’s time to think about

whether Lucene is a fit for your application.

1.3.4 Is Lucene right for your application?

As you’ve seen, a modern search application can require many components. Yet, the needs of a specific

application for each of these components vary greatly. Lucene covers many of these components (the

gray shaded ones from figure 1.4) very well, but other components are best covered by complementary

open-source software or by your own custom application logic. Or, it’s possible your application is

specialized enough to not require certain components. You should at this point have a good sense of what

we mean when we say Lucene is a search library, not a full application.

If Lucene is not a direct fit, it’s likely one of the open-source projects that complements or builds upon

Lucene does fit. For example, Solr, a sister open-source project under the Lucene Apache umbrella, adds

a server that exposes an administration interface, scaleout, indexing content from a database, and adds

important end-user functionality like faceted navigation, to Lucene. Lucene is the search library while Solr

provides most components of an entire search application. Chapter XXX covers Solr, and we also include

a case study of how Lucid Imagination uses Solr in chapter XXX. Nutch takes scaleout even further, and is

able to build indexes to handle Web-sized content collections. Projects like DBSight, Hibernate Search,

LuSQL, Compass and Oracle/Lucene make searching database content very simple by handling the

“Acquire Content” and “Build Document” steps seamlessly, as long as your content resides in a database.

Many open-source document filters exist, for deriving textual content from binary document types. Most

of the raw ingredients are there for you to pull together a powerful, fully open source search application!

Now let’s dive down and see a concrete example of using Lucene for indexing and searching.

1.4 Lucene in action: a sample application

It’s time to see Lucene in action. To do that, recall the problem of indexing and searching files, which we

described in section 1.3.1. Furthermore, suppose you need to index and search files stored in a directory

tree, not just in a single directory. To show you Lucene’s indexing and searching capabilities, we’ll use a pair

of command-line applications:

Indexer

and

Searcher

. First we’ll index a directory tree containing text

files; then we’ll search the created index.

These example applications will familiarize you with Lucene’s

API

, its ease of use, and its power. The

code listings are complete, ready-to-use command-line programs. If file indexing/searching is the problem

you need to solve, then you can copy the code listings and tweak them to suit your needs. In the chapters

that follow, we’ll describe each aspect of Lucene’s use in much greater detail.

Before we can search with Lucene, we need to build an index, so we start with our

Indexer

application.

1.4.1 Creating an index

In this section you’ll see a simple class called

Indexer

which indexes all files in a directory ending with

the

.txt

extension. When

Indexer

completes execution it leaves behind a Lucene index for its sibling,

Searcher

(presented next in section 1.4.2).

剩余398页未读，继续阅读

cjianb

粉丝: 4
资源: 11

Lucene实战：探索搜索引擎核心技术

lucene in action 第二版

lucene in action 2nd edition, lucene in action 第二版 PDF

lucene in action 第二版中文版

lucene in action 第二版英文版

Lucene in action 第二版 英文版

Lucene in Action 第二版

lucene in action第二版下载

lucene in action第二版(word版)

Lucene in Action第二版(中文和英文)

lucene in action 第二版(pdf)

最新资源

Lucene in action 第二版英文版