.
Invited Review
.
Pre-trained Models for Natural Language Processing: A Survey
Xipeng Qiu
*
, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai & Xuanjing Huang
School of Computer Science, Fudan University, Shanghai 200433, China;
Shanghai Key Laboratory of Intelligent Information Processing, Shanghai 200433, China
Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey,
we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research
progress. Then we systematically categorize existing PTMs based on a taxonomy with four perspectives. Next, we describe how to
adapt the knowledge of PTMs to the downstream tasks. Finally, we outline some potential directions of PTMs for future research.
This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.
Deep Learning, Neural Network, Natural Language Processing, Pre-trained Model, Distributed Representation, Word
Embedding, Self-Supervised Learning, Language Modelling
1 Introduction
With the development of deep learning, various neural net-
works have been widely used to solve Natural Language Pro-
cessing (NLP) tasks, such as convolutional neural networks
(CNNs) [
75
,
80
,
45
], recurrent neural networks (RNNs) [
160
,
100
], graph-based neural networks (GNNs) [
146
,
161
,
111
]
and attention mechanisms [
6
,
171
]. One of the advantages
of these neural models is their ability to alleviate the fea-
ture engineering problem. Non-neural NLP methods usually
heavily rely on the discrete handcrafted features, while neural
methods usually use low-dimensional and dense vectors (aka.
distributed representation) to implicitly represent the syntactic
or semantic features of the language. These representations
are learned in specific NLP tasks. Therefore, neural methods
make it easy for people to develop various NLP systems.
Despite the success of neural models for NLP tasks, the
performance improvement may be less significant compared
to the Computer Vision (CV) field. The main reason is that
current datasets for most supervised NLP tasks are rather small
(except machine translation). Deep neural networks usually
have a large number of parameters which make them overfit
on these small training data and do not generalize well in
practice. Therefore, the early neural models for many NLP
tasks were relatively shallow and usually consisted of only
1∼3 neural layers.
Recently, substantial work has shown that pre-trained mod-
els (PTMs) on the large corpus can learn universal language
representations, which are beneficial for downstream NLP
tasks and can avoid training a new model from scratch. With
the development of computational power, the emergence of
the deep models (i.e., Transformer [
171
]) and the constant
enhancement of training skills, the architecture of PTMs has
been advanced from shallow to deep. The first-generation
PTMs aim to learn good word embeddings. Since these mod-
els themselves are no longer needed by downstream tasks, they
are usually very shallow for computational efficiencies, such
as Skip-Gram [
116
] and GloVe [
120
]. Although these pre-
trained embeddings can capture semantic meanings of words,
they are context-free and fail to capture higher-level concepts
of text like syntactic structures, semantic roles, anaphora, etc.
* Corresponding author (email: xpqiu@fudan.edu.cn)
arXiv:2003.08271v1 [cs.CL] 18 Mar 2020
剩余24页未读，继续阅读
评论0