Question

Common applciations where there is a need to process text include:

1. Where the data is text - for example, if you are performing statistical analysis on the content of a billion web pages (perhaps you work for Google), or your research is in statistical natural language processing.
2. Where you have to preprocess messy real world dataa - e.g. column titles that are inconsistent in order to construct a DataFrame for analysis.

You may need to refer to the following:

• For string constatns and some utilitiels, see the string module - e.g string.punctuation , string.ascii_lowercase()
• For basic text processing, see string methods - e.g. lower() , upper() , split() , replace() , find() , count()
• For regulear expression use, see the re module functions, especially compile() , match() , search() , sub()

As usual, make liberal use of IPython help (e.g string.punctuation? ) to get information on a specific function or classs.

We will illustrate the use of string methods, regular expressions and natural langauge parsing, as well as some Python built-in data structures (e.g. Multiset (counter) and set) that can be used to clean or analyze text data. This is meant only as an walk-thourgh of some of the tools available; refer to the documentation for detals:

• `The string module <https://docs.python.org/2/library/string.html>>`__
• String methods
• The re module
• The regex module - new version of regular expression module
• Tutorial on regular expression
• NLTK