![](https://csdnimg.cn/release/download_crawler_static/11252710/bg12.jpg)
PREFACExvi
gained confidence, and learned more and more from my mentors and mentees, it
seemed like I might be able to build something new and magical myself.
One of the tricks I learned was to iterate through a collection of documents and
count how often words like “War” and “Hunger” are followed by words like “Games”
or “III.” If you do that for a large collection of texts, you can get pretty good at guess-
ing the right word in a “chain” of words, a phrase, or sentence. This classical approach
to language processing was intuitive to me.
Professors and bosses called this a Markov chain, but to me it was just a table of
probabilities. It was just a list of the counts of each word, based on the preceding
word. Professors would call this a conditional distribution, probabilities of words con-
ditioned on the preceding word. The spelling corrector that Peter Norvig built for
Google showed how this approach scales well and takes very little Python code.
2
All
you need is a lot of natural language text. I couldn’t help but get excited as I thought
about the possibilities for doing such a thing on massive free collections of text like
Wikipedia or the Gutenberg Project.
3
Then I heard about latent semantic analysis (LSA). It seemed to be just a fancy way
of describing some linear algebra operations I’d learned in college. If you keep track
of all the words that occur together, you can use linear algebra to group those words
into “topics.” LSA could compress the meaning of an entire sentence or even a long
document into a single vector. And, when used in a search engine, LSA seemed to
have an uncanny ability to return documents that were exactly what I was looking for.
Good search engines would do this even when I couldn’t think of the words that
might be in those documents!
Then
gensim
released a Python implementation of Word2vec word vectors, mak-
ing it possible to do semantic math with individual words. And it turned out that this
fancy neural network math was equivalent to the old LSA technique if you just split up
the documents into smaller chunks. This was an eye-opener. It gave me hope that I
might be able to contribute to the field. I’d been thinking about hierarchical semantic
vectors for years—how books are made of chapters of paragraphs of sentences of
phrases of words of characters. Tomas Mikolov, the Word2vec inventor, had the insight
that the dominant semantics of text could be found in the connection between two
layers of the hierarchy, between words and 10-word phrases. For decades, NLP
researchers had been thinking of words as having components, like niceness and emo-
tional intensity. And these sentiment scores, components, could be added and sub-
tracted to combine the meanings of multiple words. But Mikolov had figured out how
to create these vectors without hand-crafting them, or even defining what the compo-
nents should be. This made NLP fun!
2
See the web page titled “How to Write a Spelling Corrector” by Peter Norvig (http://www.norvig.com/spell-
correct.html).
3
If you appreciate the importance of having freely accessible books of natural language, you may want to keep
abreast of the international effort to extend copyrights far beyond their original
“use by” date: gutenberg.org
(http://www.gutenberg.org) and gutenbergnews.org (http://www.gutenbergnews.org/20150208/copyright-
term-extensions-are-looming:).