A Primer on Neural Network Models
for Natural Language Processing
Yoav Goldberg
Draft as of October 6, 2015.
The most up-to-date version of this manuscript is available at http://www.cs.biu.
ac.il/
˜
yogo/nnlp.pdf. Major updates will be published on arxiv periodically.
I welcome any comments you may have regarding the content and presentation. If you
spot a missing reference or have relevant work you’d like to see mentioned, do let me know.
first.last@gmail
Abstract
Over the past few years, neural networks have re-emerged as powerful machine-learning
models, yielding state-of-the-art results in fields such as image recognition and speech
processing. More recently, neural network models started to be applied also to textual
natural language signals, again with very promising results. This tutorial surveys neural
network models from the perspective of natural language processing research, in an attempt
to bring natural-language researchers up to speed with the neural techniques. The tutorial
covers input encoding for natural language tasks, feed-forward networks, convolutional
networks, recurrent networks and recursive networks, as well as the computation graph
abstraction for automatic gradient computation.
1. Introduction
For a long time, core NLP techniques were dominated by machine-learning approaches that
used linear models such as support vector machines or logistic regression, trained over very
high dimensional yet very sparse feature vectors.
Recently, the field has seen some success in switching from such linear models over
sparse inputs to non-linear neural-network models over dense inputs. While most of the
neural network techniques are easy to apply, sometimes as almost drop-in replacements of
the old linear classifiers, there is in many cases a strong barrier of entry. In this tutorial I
attempt to provide NLP practitioners (as well as newcomers) with the basic background,
jargon, tools and methodology that will allow them to understand the principles behind
the neural network models and apply them to their own work. This tutorial is expected
to be self-contained, while presenting the different approaches under a unified notation and
framework. It repeats a lot of material which is available elsewhere. It also points to
external sources for more advanced topics when appropriate.
This primer is not intended as a comprehensive resource for those that will go on and
develop the next advances in neural-network machinery (though it may serve as a good entry
point). Rather, it is aimed at those readers who are interested in taking the existing, useful
technology and applying it in useful and creative ways to their favourite NLP problems. For
more in-depth, general discussion of neural networks, the theory behind them, advanced
1
arXiv:1510.00726v1 [cs.CL] 2 Oct 2015
剩余74页未读，继续阅读
评论1