Natural Language Toolkit

Article Info

Contributed by
2 authors

Last updated on
2019-10-28 02:19:44

Natural Language Processing
Text Corpus for NLP
spaCy
Gensim
TextBlob
Apache OpenNLP

Article Versions

12 2019-10-28 02:19:44
1716,1715 12,1716

By arvindpdmn

Summary done. Publishing.
11 2019-10-27 15:27:07
1715,1714 11,1715

By arvindpdmn

Changed order of questions. Uploaded images of better resolution. Citation corrected for one figure.
10 2019-10-27 14:58:54
1714,1713 10,1714

By arvindpdmn

Adding Milestones, Further Reading, See Also. Updating Discussion. Summary pending.
9 2019-10-27 10:24:09
1713,1712 9,1713

By arvindpdmn

Added limitations question.
8 2019-10-27 10:16:32
1712,1711 8,1712

By pabitra_data

update- modules language dataset added, references to be rectified

Chat Room

Submitting ...

You are editing an existing chat message.
2019-10-28 03:16:45
-

By arvindpdmn

For milestones, the original paper helps. Plus release notes, GitHub record of releases, etc.

Your research is good for a first article. Your next article will be even better. The software you use for capturing screenshots is giving poor quality images. Prefer to save the original images featured on webpages.
2019-10-28 02:24:00
-

By pabitra_data

I was just going to write the summary
Good that you completed it.
Now, its publishable i guess.

I can evaluate that my research needs to be more intense for better quality of writing.

Also, i would like to ask how do you approach for milestones ?

I would like to take aspect mining next as i have some idea on it.
2019-10-27 14:59:38
-

By arvindpdmn

Updated the refs you mentioned. Please complete the Summary and then I will publish.
2019-10-27 10:17:56
-

By pabitra_data

Please help me on the references in last 2 questions
i am not able to find names and other details
2019-10-17 03:15:08
-

By arvindpdmn

Excellent questions. I feel there may be duplication with some questions. When you fill in the answers you can take a call. Some questions may be too detailed for a high-level article but that's okay. We can move them later into a separate article if this one gets too long.

Talk about industry adoption of NLTK, performance, problems or limitations.

Do a Google image search. This may give you more ideas about how to organize the content.

Will be good to give an overview of important NLTK modules in a separate question. Here's a complete list based on 3.4.5: app, ccg, chat, chunk, classify, cluster, corpus, draw, inference, lm, metrics, misc, parse, sem, sentiment, stem, tag, tbl, test, tokenize, translate, twitter.

Typical NLTK pipeline for information extraction. Source: Bird et al. 2019, ch. 7, fig. 7.1.

Natural Language Toolkit (NLTK) is a Python package to perform natural language processing (NLP). It was created mainly as a tool for learning NLP via a hands-on approach. It was not designed to be used in production.

The growth of unstructured data via social media, online reviews, blogs, and voice-based human-computer interaction are some reasons why NLP has become important in the late 2010s. NLTK is a useful toolkit for many of these NLP applications.

NLTK is composed of sub-packages and modules. A typical processing pipeline will call modules in sequence. Python data structures are passed from one module to another. Beyond the algorithms, NLTK gives quick access to many text corpora and datasets.

Discussion

Which are the fundamental NLP tasks that can be performed using NLTK?
Pipeline for text classification. Source: Navlani 2018, fig. 1.
NLTK can be used in wide range of applications for NLP. For basic understanding, let's try to analyze a paragraph using NLTK. It can be pre-processed using sentence segmentation, removing stopwords, removing punctuation and special symbols, and word tokenization. After pre-processing the corpus, it can be analyzed sentence-wise using parts of speech (POS) to extract nouns and adjectives. Subsequent tasks can include named entity recognition (NER), coreference resolution, constituency parsing and dependency parsing. The goal is to find insights and context about the corpus.
Further downstream tasks, more pertaining to application areas, could be emotion detection, sentiment analysis or text summarization. Tasks such as text classification and topic modeling typically require large amounts of text for better results.
Which are the modules available in NLTK?
NLTK modules with functionalities. Source: Bird et al. 2019, ch. 0, table VIII.1.
NLTK's architecture is modular. Functionality is organized into sub-packages and modules. NLTK is used for its simplicity, consistency and extensibility of its modules and functions. It's better explained in the tabular list of modules.
A complete module index is available as part of NLTK documentation.
How is NLTK package split into sub-packages and modules?
Illustrating the organization of 'text' sub-package and its modules. Source: Howard 2016, fig. 3.
NLTK is divided into different sub-packages and modules for text analysis using various methods. Figure depicts an example of text sub-package and the modules within it. Each module fulfils a specific function.
Which are the natural languages supported in NLTK?
Languages supported by NLTK depends on the task being implemented. For stemming, we have RSLPStemmer (Portuguese), ISRIStemmer (Arabic), and SnowballStemmer (Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish).
For sentence tokenization, PunktSentenceTokenizer is capable of multilingual processing.
Stopwords are also available in multiple languages. After importing stopwords, we can obtain a list of languages by running print(stopwords.fileids()).
Although most taggers in NLTK support only English, nltk.tag.stanford module allows us to use StanfordPOSTagger, which has multilingual support. This is only an interface to the Stanford tagger, which must be running on the machine.
What datasets are available in NLTK for practice?
NLTK downloader is a handy interface to manage packages and datasets. Source: Shetty 2018.
NLTK corpus is a natural dump for all kinds of NLP datasets that can be used for practice or maybe combined for generating models. For example, to import the Inaugural Address address, the statement to execute is from nltk.corpus import inaugural.
Out of dozens of corpora, some popular ones are Brown, Name Genders, Penn Treebank, and Inaugural Address. NLTK makes it easy to read a corpus via the package nltk.corpus. This package has a reader object for each corpus.
What are the disadvantages or limitations of NLTK?
It's been mentioned that NLTK is "a complicated solution with a harsh learning curve and a maze of internal limitations".
For sentence tokenization, NLTK doesn't apply semantic analysis. Unlike Gensim, NLTK lacks neural network models or word embeddings.
NLTK is slow, whereas spaCy is said to be the fastest alternative. In fact, since NLTK was created educational purpose, optimized runtime performance was never a goal. However, it's possible to speed up execution using Python's multiprocessing module.
Matthew Honnibal, the creator of spaCy, noted that NTLK has lots of modules but very few (tokenization, stemming, visualization) are actually useful. Often NLTK has wrappers to external libraries and this leads to slow execution. The POS tagger was terrible, until Honnibal's averaged perceptron tagger was merged into NLTK in September 2015.
In general, NLP is evolving so fast that maintainers need to curate often and throw away old things.
For beginners, what are some useful resources to learn NLTK?
The official website includes documentation, Wiki, and index of all modules. There are Google Groups for users and developers.
For basic usage of NLTK, you can read a tutorial by Bill Chambers. This also shows some text classification examples using Scikit-learn. Another basic tutorial from Harry Howard includes examples from Pattern library as well.
Often specific processing is implemented in external libraries. Benjamin Bengfort shows in a blog post how to call CoreNLP from inside NLTK for syntactic parsing.
There's a handy cheat sheet by murenei. Another one from 2017 is published at Northwestern University.
A list of recommended NLTK books appears on BookAuthority. You can start by reading Natural Language Processing with Python (Bird et al. 2009). Those who wish to learn via videos can look up a playlist of 21 videos from sentdex.

Milestones

Jul
2001

NLTK's chart parsing tool is a useful visualization. Source: Loper and Bird 2002, fig. 1.

The first downloadable version of NLTK appears on SourceForge. Created at the University of Pennsylvania, the aim is to have a set of open source software, tutorials and problem sets to aid the teaching of computational linguistics. Before NLTK, a project might require students to learn multiple programming languages and toolkits. Lack of visualizations also made it difficult to have class demonstrations. NLTK is meant to solve these problems.

Jul
2005

NLTK-Lite 0.1 is released. Steven Bird, one of the creators of NLTK, explains that NLTK 1.4 introduced Python's dictionary-based architecture for storing tokens. This created overhead for programmers. With NLTK-Lite, programmers can use simpler data structures. For better performance, iterators are used instead of lists. Taggers use backoff by default. Method names are shorter. Since then, regular releases are made till NLTK-Lite 0.9 in October 2007. NLTK-Lite eventually becomes NLTK.

Apr
2008

Two NLTK projects are accepted for Google Summer of Code: dependency parsing and natural language generation. The dependency parser becomes part of NLTK version 0.9.6 (December 2008).

Jun
2009

Book titled Natural Language Processing with Python by Bird et al. is published by O'Reilly Media. Since October 2013, the authors release online revised versions of the book updated for Python 3 and NLTK 3.

Apr
2011

Version 2.0.1rc1 becomes the first release available via GitHub, although till July 2014 releases are also made via SourceForge.

Jul
2013

Over a five-year period from January 2008 to July 2013, NLTK gets more than half a million downloads. This excludes downloads via GitHub.

Sep
2014

NLTK 3.0.0 is released, making this the first stable release supporting Python 3. Alpha release, version 3.0a0 (alpha), supporting Python 3 can be traced to January 2013.