改进的语义相似度向量空间模型聚焦爬虫技术
191 浏览量
更新于2024-07-14
收藏 1.46MB PDF 举报
“基于语义相似度向量空间模型的改进型聚焦爬虫”
这篇研究论文“An improved focused crawler based on Semantic Similarity Vector Space Model”探讨了如何利用语义相似度和向量空间模型（VSM）来改进聚焦爬虫的性能。聚焦爬虫是一种专门针对特定主题进行网页抓取的爬虫，它旨在从互联网上筛选出与给定主题相关的页面。
在当前的搜索引擎和信息检索领域，向量空间模型（VSM）是计算文档之间相似度的常用方法。该模型将文本转换为向量，通过计算两个向量之间的余弦相似度来评估它们的相关性。然而，VSM在处理未共享词汇（即词汇鸿沟问题）时可能遇到挑战，这可能导致相关网页的漏抓或非相关网页的误抓。
为了解决这个问题，研究引入了语义相似度检索模型（SSRM）。SSRM不仅考虑词频，还考虑词语的上下文和语义关系，以提高相似度计算的准确性。它通常利用词嵌入技术，如Word2Vec或GloVe，来捕获词汇的深层语义信息。这种方法使得即使两个词在表面形式上不同，但若它们在语义上相似，也能被识别出来。
论文指出，通过结合VSM和SSRM，可以创建一个更强大的聚焦爬虫，它能更有效地识别和抓取与目标主题相关的网页。这种改进的爬虫策略可能会减少无关页面的抓取，提高爬取效率，并优化主题相关性的判断。
关键词：聚焦爬虫、语义相似度、VSM、SSRM
文章历史：2014年2月17日提交，2015年3月27日修订，2015年7月15日接受，2015年8月1日在线发布。
这篇研究论文对改进聚焦爬虫进行了深入探讨，提出了利用语义相似度增强的向量空间模型，以解决传统方法在处理语义关联时的不足，从而提升网络爬取的精确性和效率。这一研究成果对于搜索引擎优化、信息检索以及大数据分析等领域具有重要的理论和实践价值。
Y.
Du
et
al.
/
Applied
Soft
Computing
36
(2015)
392–407
395
The
FCA
is
focused
on
the
following:
FCA
computes
seman-
tic
similarities
between
terms
by
using
extension
similarities
and
intension
similarities
of
formal
concepts
[16].
FCA
first
considers
each
term
to
be
an
object,
and
considers
different
semantics
of
this
term
to
be
attributes.
Next,
these
terms
and
their
semantics
are
used
to
construct
different
formal
concepts
and
a
corre-
sponding
concept
lattice.
Additionally,
the
extension
similarity
between
the
formal
concepts
is
computed
based
on
the
relation-
ships
between
the
extensions
of
the
concepts
in
the
ontology,
while
the
intension
similarity
is
computed
based
on
the
relationships
between
the
intension
of
the
concepts
in
the
concept
lattice.
Finally,
the
semantic
similarity
between
two
terms
is
computed
by
lin-
early
integrating
the
extension
similarity
and
intension
similarity
between
two
concepts.
3.
Improved
focused
crawler
based
on
Semantic
Similarity
Vector
Space
Model
In
this
section,
the
prediction
of
the
priorities
for
unvisited
hyperlinks
by
the
SSVSM,
as
a
guide
for
the
improved
focused
crawler
to
retrieve
topic-relevant
web
pages,
is
discussed.
Fig.
1
shows
the
flowchart
for
an
improved
focused
crawler
based
on
SSVSM.
The
flowchart
for
this
focused
crawler
is
divided
into
three
main
modules,
as
follows:
acquiring
term
weights
and
semantic
similarity
matrixes,
computing
topical
similarities
of
the
full
texts
and
anchor
texts
based
on
SSVSM
and
designing
priorities
for
unvis-
ited
hyperlinks.
First,
this
focused
crawler
parses
the
hyperlinks
from
the
crawled
web
pages
in
a
web
page
database,
and
places
these
hyperlinks
in
the
unvisited
list.
Correspondingly,
the
full
texts
and
anchor
texts
of
each
unvisited
hyperlink
are
extracted
from
the
web
pages.
Then,
the
term
weights
of
these
text
segments
are
acquired
based
on
the
TF*IDF
formula,
and
semantic
similar-
ity
matrixes
between
the
terms
of
these
texts
and
the
terms
of
the
topic
are
calculated
based
on
formula
(5).
Second,
the
topical
simi-
larities
of
the
full
texts
and
anchor
texts
of
the
unvisited
hyperlinks
are
computed
based
on
the
SSVSM.
The
SSVSM
constructs
semantic
vectors
that
are
mapped
to
the
same
semantic
space
for
the
topic
and
the
above
two
text
segments.
The
cosine
similarity
between
the
topic
and
the
semantic
vectors
of
each
text
segment
is
computed
as
the
topic-relevant
similarity
of
the
text.
Finally,
the
two
topi-
cal
similarities
of
the
full
texts
and
anchor
texts
of
each
unvisited
hyperlink
are
linearly
integrated
as
the
priority
of
the
hyperlink.
To
continue
downloading
web
pages
from
the
Internet,
this
focused
crawler
selects
unvisited
hyperlinks
that
have
higher
priorities.
3.1.
Term
weights
and
semantic
similarity
matrixes
Terms
are
extracted
from
the
given
topic
and
documents
of
the
hyperlinks
to
predict
the
priority
of
an
unvisited
hyperlink
that
is
related
to
a
given
topic.
Documents
of
hyperlinks
include
many
dif-
ferent
text
segments.
In
this
paper,
the
documents
of
the
unvisited
hyperlinks
are
classified
into
full
texts
and
anchor
texts
from
the
web
pages.
More
importantly,
the
full
text
terms
and
anchor
text
terms
are
considered
to
be
text
terms
of
hyperlinks
to
more
accu-
rately
predict
their
priorities
for
the
hyperlinks
that
are
related
to
the
given
topic.
This
procedure
primarily
computes
the
term
weights
and
semantic
similarity
matrixes
for
the
given
topic,
full
texts
and
anchor
texts.
The
given
topic
can
be
manually
provided
with
a
set
of
topic-relevant
web
pages.
The
full
text
is
derived
from
the
contents
of
the
retrieved
web
pages.
In
contrast,
the
anchor
text
could
come
from
various
retrieved
web
pages,
and
the
description
of
a
hyperlink
in
a
single
web
page
is
usually
very
short.
Therefore,
all
of
the
descriptions
of
the
hyperlink
are
wholly
integrated
as
the
anchor
text.
For
computing
the
term
weights,
there
are
two
major
components,
as
follows:
(1)
Pre-processing
the
given
topic,
full
texts
and
anchor
texts.
This
component
includes
term
stopping
and
term
stemming.
The
term
stopping
module
removes
the
most
common
words,
such
as
‘the’,
‘of’,
and
‘for’,
which
contribute
little
to
the
topics
of
the
web
pages.
The
term
stemming
function
groups
words
that
are
derived
from
the
same
stem,
base,
or
root
form
[33].
For
exam-
ple,
the
‘tree’,
‘trees’
and
‘treeing’
can
be
organized
as
‘tree’.
After
this
pre-processing,
many
topic
terms
and
text
terms
from
the
given
topic,
full
texts
and
anchor
texts
can
be
easily
extracted.
(2)
Calculating
the
term
weights
for
the
given
topic,
full
texts
and
anchor
texts.
The
term
weights
can
quantitatively
express
the
whole
document
with
long
characters.
The
term
weights
are
commonly
computed
based
on
formula
(3).
The
semantic
similarity
between
two
arbitrary
terms
can
be
computed
by
employing
their
relationships
in
the
ontology.
In
this
section,
the
semantic
similarities
between
the
topic
terms
and
text
terms
are
computed
based
on
formula
(5).
Specifically,
the
text
terms
are
classified
into
full
text
terms
and
anchor
text
terms.
Cor-
respondingly,
the
semantic
similarities
between
the
topic
and
text
terms
form
two
different
matrixes,
including
the
full
text–topic
matrix
and
the
anchor
text–topic
matrix.
In
addition,
the
text
terms
represent
the
rows
of
the
matrixes,
while
the
topic
terms
repre-
sent
the
columns
of
the
matrixes.
These
two
semantic
similarity
matrixes
are
described
in
detail
as
follows:
term
1
of
topic
t
term
2
of
topic
t
·
·
·
term
m
of
topic
t
term
1
of
f
p
Sem
p
11
Sem
p
12
·
·
·
Sem
p
1m
term
2
of
f
p
Sem
p
21
Sem
p
22
·
·
·
Sem
p
2m
.
.
.
.
.
.
.
.
. ·
·
·
.
.
.
term
n
of
f
p
Sem
p
n1
Sem
p
n2
·
·
·
Sem
p
nm
(6)
term
1
of
topic
t
term
2
of
topic
t
·
·
·
term
m
of
topic
t
term
1
of
a
l
Sem
l
11
Sem
l
12
·
·
·
Sem
l
1m
term
2
of
a
l
Sem
l
22
Sem
l
22
·
·
·
Sem
l
2m
.
.
.
.
.
.
.
.
.
·
·
·
.
.
.
term
r
of
a
l
Sem
l
r1
Sem
l
r2
·
·
·
Sem
l
rm
(7)
where
f
p
,
a
l
,
t
(1
≤
p
≤
P,
1
≤
l
≤
L)
are,
respectively,
the
full
text
of
the
retrieved
web
page
p,
the
anchor
text
of
the
unvisited
hyper-
link
l
and
the
given
topic;
P,
L
are,
respectively,
the
total
numbers
of
the
retrieved
web
pages
and
the
unvisited
hyperlinks;
term
i
of
f
p
,
term
k
of
a
l
,
term
j
of
topic
t
(1
≤
i
≤
n,
1
≤
k
≤
r,
1
≤
j
≤
m)
are,
respec-
tively,
the
term
i
of
the
full
text
f
p
,
the
term
k
of
the
anchor
text
a
l
and
the
term
j
of
the
topic
t;
Sem
p
ij
is
the
semantic
similarity
between
term
i
of
f
p
and
term
j
of
topic
t;
Sem
l
kj
is
the
semantic
similar-
ity
between
term
k
of
a
l
and
term
j
of
topic
t;
m,
n,
r
are,
respectively,
the
number
of
topic
terms,
the
number
of
full
text
terms
of
the
retrieved
web
page
p
and
the
number
of
anchor
text
terms
of
the
unvisited
hyperlink
l.
3.2.
Semantic
Similarity
Vector
Space
Model
In
this
module,
the
Semantic
Similarity
Vector
Space
Model
(SSVSM)
is
described
in
detail.
The
SSVSM
can
detect
similarities
剩余15页未读，继续阅读
117 浏览量
2009-07-31 上传
点击了解资源详情
183 浏览量
2021-04-14 上传
2023-02-23 上传
651 浏览量
102 浏览量
651 浏览量
weixin_38587705
- 粉丝: 0
- 资源: 930
我的内容管理
展开
最新资源
- kubernetes-kms:for适用于Kubernetes的Azure Key Vault KMS插件
- Data_Explore_py_pandas_Professional_nanodegree_program：具有一些基本描述性统计信息的用户交互式数据探索程序
- IntelligentAgentsAssignment:第一次尝试在非常简单的环境中实现信念-愿望-意图模型
- flash元件批量改名命令（jsfl）
- fullstackopen:赫尔辛基大学
- Calendar2.rar
- vscode-mono-debug:一个简单的VS Code调试适配器，用于单声道
- packtools:用于处理SciELO PS XML文件的Python库和命令行实用程序
- 使用 MATLAB 进行信用风险建模：这些是 MathWorks 网络研讨会的同名 MATLAB 支持文件。-matlab开发
- 采购管理工程招投标流程
- CBB-Stats
- 12.XGBoost_data.rar
- 电子功用-基于电压跟踪的锂电池剩余电量的计量方法
- 皇家型
- android:android相关代码和示例
- 采购与仓储管理
