6180
Center in New York was...
B-PER
Source 1
O O B-LOC I-LOC
O...
B-LOC
Source 2
I-LOC O O B-LOC
O...
B-LOC
Target
I-LOC O I-LOC
O...
B-LOC
Rockefeller
Figure 1: An example of label aggregation with two
weak labeling sources. We use BIO labeling scheme.
PER represents person; LOC is location.
mizes the marginal likelihood of the observations.
3 Problem Setup
In this section, we formulate the multi-source
weakly supervised NER problem. Consider an in-
put sentence that contains
T
tokens
w
(1:T )
, NER
can be formulated as a sequence labeling task that
assigns a label to each token in the sentence.
1
As-
suming the set of target entity types is
E
and the tag-
ging scheme is BIO (Ramshaw and Marcus, 1995),
NER models assign one label from the label set
l ∈ L
to each token, where the size of the label set
is
|L| = 2|E| + 1
, e.g., if
E = {PER, LOC}
, then
L = {O, B-PER, I-PER, B-LOC, I-LOC}.
Suppose we have a sequence with
K
weak
sources, each of which can be a heuristic rule,
knowledge base, or existing out-of-domain NER
model. Each source serves as a labeling function
that generates token-level weak labels from the in-
put corpus, as shown in Figure 1. For the input se-
quence
w
(1:T )
, we use
x
(1:T )
k
, k ∈ {1, . . . , K}
to
represent the weak labels from the source
k
, where
x
(t)
k
∈ R
|L|
, t ∈ {1, . . . , T }
is a probability dis-
tribution over
L
. Multi-source weakly supervised
NER aims to find the underlying true sequence of
labels
ˆ
y
(1:T )
, ˆy
(t)
∈ L given {w
(1:T )
, x
(1:T )
1:K
}.
4 Methodology
In this section, we describe our proposed method
CHMM-ALT. We first sketch the alternate-training
procedure (
§
4.1), then explain the CHMM com-
ponent (
§
4.2) and how BERT-NER is involved
(§ 4.3).
4.1 Alternate-Training Procedure
The alternate-training method trains two models—
a multi-source label aggregator CHMM and a
BERT-NER model—in turn with each other’s out-
put. CHMM aggregates multiple sets of labels
from different sources into a unified sequence of
1
We represent vectors, matrices or tensors with bold fonts
and scalars with regular fonts; 1 : a , {1, 2, . . . , a}.
labels, while BERT-NER refines them by its lan-
guage modeling ability gained from pre-training.
The training process is divided into two phases.
•
In
phase I
, CHMM takes the annotations
x
(1:T )
1:K
from existing sources and gives a set
of denoised labels
y
∗
(1:T )
, which are used
to fine-tune the BERT-NER model. Then,
we regard the fine-tuned model as an addi-
tional labeling source, whose outputs
˜
y
(1:T )
are added into the original weak label sets
to give the updated observation instances:
x
(1:T )
1:K+1
= {x
(1:T )
1:K
,
˜
y
(1:T )
}.
•
In
phase II
, CHMM and BERT-NER mutu-
ally improve each other iteratively in several
loops. Each loop first trains CHMM with the
observation
x
(1:T )
1:K+1
from the previous one.
Then, its predictions are adopted to fine-tune
BERT-NER, whose output updates x
(1:T )
K+1
.
Figure 2 illustrates the alternate-training method.
In general, CHMM gives high precision predic-
tions, whereas BERT-NER trades recall with preci-
sion. In other words, CHMM can classify named
entities with high accuracy but is slightly disadvan-
taged in discovering all entities. BERT-NER in-
creases the coverage with a certain loss of accuracy.
Combined with the alternate-training approach, this
complementarity between these models further in-
creases the overall performance.
4.2 Conditional Hidden Markov Model
The conditional hidden Markov model is an HMM
variant for multi-source label denoising. It mod-
els true entity labels as hidden variables and infers
them from the observed noisy labels. Tradition-
ally, discrete HMM uses one transition matrix to
model the probability of hidden label transitioning
and one emission matrix to model the probability
of the observations from the hidden labels. These
two matrices are constant, i.e., their values do not
change over time steps. CHMM, on the contrary,
conditions both its transition and emission matri-
ces on the BERT embeddings
e
(1:T )
of the input
tokens
w
(1:T )
. This design not only allows CHMM
to leverage the rich contextual representations of
the BERT embeddings but relieves the constant
matrices constraint as well.
In phase I, CHMM takes
K
sets of weak labels
from the provided
K
weak labeling sources. In
phase II, in addition to the existing sources, it takes
剩余12页未读，继续阅读
dududuv
- 粉丝: 0
- 资源: 7
我的内容管理
展开
最新资源
- 解决Eclipse配置与导入Java工程常见问题
- 真空发生器：工作原理与抽吸性能分析
- 爱立信RBS6201开站流程详解
- 电脑开机声音解析：故障诊断指南
- JAVA实现贪吃蛇游戏
- 模糊神经网络实现与自学习能力探索
- PID型模糊神经网络控制器设计与学习算法
- 模糊神经网络在自适应PID控制器中的应用
- C++实现的学生成绩管理系统设计
- 802.1D STP 实现与优化：二层交换机中的生成树协议
- 解决Windows无法完成SD卡格式化的九种方法
- 软件测试方法：Beta与Alpha测试详解
- 软件测试周期详解：从需求分析到维护测试
- CMMI模型详解：软件企业能力提升的关键
- 移动Web开发框架选择：jQueryMobile、jQTouch、SenchaTouch对比
- Java程序设计试题与复习指南
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
