![](https://csdnimg.cn/release/download_crawler_static/23385553/bg3.jpg)
6180
Center in New York was...
B-PER
Source 1
O O B-LOC I-LOC
O...
B-LOC
Source 2
I-LOC O O B-LOC
O...
B-LOC
Target
I-LOC O I-LOC
O...
B-LOC
Rockefeller
Figure 1: An example of label aggregation with two
weak labeling sources. We use BIO labeling scheme.
PER represents person; LOC is location.
mizes the marginal likelihood of the observations.
3 Problem Setup
In this section, we formulate the multi-source
weakly supervised NER problem. Consider an in-
put sentence that contains
T
tokens
w
(1:T )
, NER
can be formulated as a sequence labeling task that
assigns a label to each token in the sentence.
1
As-
suming the set of target entity types is
E
and the tag-
ging scheme is BIO (Ramshaw and Marcus, 1995),
NER models assign one label from the label set
l ∈ L
to each token, where the size of the label set
is
|L| = 2|E| + 1
, e.g., if
E = {PER, LOC}
, then
L = {O, B-PER, I-PER, B-LOC, I-LOC}.
Suppose we have a sequence with
K
weak
sources, each of which can be a heuristic rule,
knowledge base, or existing out-of-domain NER
model. Each source serves as a labeling function
that generates token-level weak labels from the in-
put corpus, as shown in Figure 1. For the input se-
quence
w
(1:T )
, we use
x
(1:T )
k
, k ∈ {1, . . . , K}
to
represent the weak labels from the source
k
, where
x
(t)
k
∈ R
|L|
, t ∈ {1, . . . , T }
is a probability dis-
tribution over
L
. Multi-source weakly supervised
NER aims to find the underlying true sequence of
labels
ˆ
y
(1:T )
, ˆy
(t)
∈ L given {w
(1:T )
, x
(1:T )
1:K
}.
4 Methodology
In this section, we describe our proposed method
CHMM-ALT. We first sketch the alternate-training
procedure (
§
4.1), then explain the CHMM com-
ponent (
§
4.2) and how BERT-NER is involved
(§ 4.3).
4.1 Alternate-Training Procedure
The alternate-training method trains two models—
a multi-source label aggregator CHMM and a
BERT-NER model—in turn with each other’s out-
put. CHMM aggregates multiple sets of labels
from different sources into a unified sequence of
1
We represent vectors, matrices or tensors with bold fonts
and scalars with regular fonts; 1 : a , {1, 2, . . . , a}.
labels, while BERT-NER refines them by its lan-
guage modeling ability gained from pre-training.
The training process is divided into two phases.
•
In
phase I
, CHMM takes the annotations
x
(1:T )
1:K
from existing sources and gives a set
of denoised labels
y
∗
(1:T )
, which are used
to fine-tune the BERT-NER model. Then,
we regard the fine-tuned model as an addi-
tional labeling source, whose outputs
˜
y
(1:T )
are added into the original weak label sets
to give the updated observation instances:
x
(1:T )
1:K+1
= {x
(1:T )
1:K
,
˜
y
(1:T )
}.
•
In
phase II
, CHMM and BERT-NER mutu-
ally improve each other iteratively in several
loops. Each loop first trains CHMM with the
observation
x
(1:T )
1:K+1
from the previous one.
Then, its predictions are adopted to fine-tune
BERT-NER, whose output updates x
(1:T )
K+1
.
Figure 2 illustrates the alternate-training method.
In general, CHMM gives high precision predic-
tions, whereas BERT-NER trades recall with preci-
sion. In other words, CHMM can classify named
entities with high accuracy but is slightly disadvan-
taged in discovering all entities. BERT-NER in-
creases the coverage with a certain loss of accuracy.
Combined with the alternate-training approach, this
complementarity between these models further in-
creases the overall performance.
4.2 Conditional Hidden Markov Model
The conditional hidden Markov model is an HMM
variant for multi-source label denoising. It mod-
els true entity labels as hidden variables and infers
them from the observed noisy labels. Tradition-
ally, discrete HMM uses one transition matrix to
model the probability of hidden label transitioning
and one emission matrix to model the probability
of the observations from the hidden labels. These
two matrices are constant, i.e., their values do not
change over time steps. CHMM, on the contrary,
conditions both its transition and emission matri-
ces on the BERT embeddings
e
(1:T )
of the input
tokens
w
(1:T )
. This design not only allows CHMM
to leverage the rich contextual representations of
the BERT embeddings but relieves the constant
matrices constraint as well.
In phase I, CHMM takes
K
sets of weak labels
from the provided
K
weak labeling sources. In
phase II, in addition to the existing sources, it takes