Natural Language Processing (NLP) Fundamentals

April 3, 2016 4 Comments

This is an introductory post on Natural Language Processing (NLP) Fundamentals. For topic-specific content, please find the following list of contents:

1. Finite State Transducers (FSTs)
2. Hidden Markov Models (HMMs) [Part 1] [Part 2]
3. Maximum Entropy (MaxEnt)
4. Maximum Entropy Markov Models (MEMMs)*
5. Conditional Random Fields*

* To be published

Disclaimer: I am a developer at Amazon.com for their Alexa technologies, on devices such as Amazon Echo. As a consequence, there may be hints of Amazon sprinkled throughout this blog series. However, I should note that this blog series is not sponsored by Amazon.com, and represent my opinions only, and not those of the company.

Relative to buzzwords like Big Data, Machine Learning, or Deep Neural Networks, Natural Language Processing is not a particularly well-known field outside of STEM professionals, but has enjoyed pervasive applications. Ever wonder how email providers (like Gmail, Outlook, or Hotmail) automatically determine whether an email is spam? When someone submits a product review about a pair of sneakers, how do computers figure out if the review speaks positively or negatively about attributes like style, color, or fit? If I ask Amazon Echo a command such as “Alexa, what’s the weather today?”, how does Alexa figure out what I meant, and determine how to answer?

Natural Language Processing (NLP) helps explain how these systems (and more!) operate. NLP is comprised of concepts from linguistics, computer science, and machine learning.

I have found myself personally interested in NLP. As a software development engineer for Amazon’s Alexa technology, I work closely with machine learning scientists to help them automate their workflows and release processes. It helps to understand the theories that form the basis of their work, and at the core of it all is NLP.

Over the past few months, I studied from a number of resources that I have found to be super useful for getting my feet wet in NLP:

• Language and Speech Processing, 2nd Edition, Jurafsky and Martin
• “Natural Language Processing”, taught by Jurafsky and Manning (Stanford University @ Coursera)
• Log-Linear Models, MEMMs, and CRFs, by Collins
• An Introduction to Conditional Random Fields for Relational Learning, by Sutton and McCallum

This blog series, NLP Fundamentals, will be a collection of explanations of different NLP concepts. These explanations will be curated from the resources above, inspired from other resources, and also self-generated to accelerate my understanding.

Throughout the series, I will be writing about the following concepts:

1. Finite State Transducers (FSTs)
2. Hidden Markov Models (HMMs) [Part 1] [Part 2]
3. Maximum Entropy (MaxEnt)
4. Maximum Entropy Markov Models (MEMMs)
5. Conditional Random Fields

This is hardly a comprehensive survey of topics in NLP (I don’t know it all — in fact, I’m still exploring!), and more closely resembles a summary-level view of selected topics in NLP. I’ll try my best to provide more resources for each topic for further reading or experimentation.

Why would you be interested in reading these posts?

• I intend to provide sample problems and code along the way, so on top of learning by explanation, you will be able to learn through examples.
• I’m a complete noob, but I now have a pretty firm, basic understanding of these concepts. You might benefit from the intuitions I’ve derived for each concept.
• I’m a biomedical engineer (with an electrical engineering Ph.D. minor) from academia, and a software development engineer from the industry. I’m not a machine learning scientist, however cool as that title is. As a result, I will be writing from a background that you might be able to relate with, particularly if you are coming from a STEM background that is not firmly rooted in machine learning/NLP.

Natural Language Processing Conditional Random Fields, CRFs, Finite State Transducers, FSTs, Hidden Markov Models, HMMs, Machine Learning, MaxEnt, Maximum Entropy, Maximum Entropy Markov Models, MEMMs