Statistical NLP Approach for Collocation Extraction from Newspaper Text

Statistical NLP is an approach of doing natural language processing to resolve the problems usually encountered with traditional NLP. The task of finding some interesting combination of wordsfrom text in large corpora, known as collocation extraction, is one of many tasks in statistical NLP. Collocations have multiple applications and the methods of collocation extraction are influenced by their intended use. In the present task, the effectiveness of selecting bigram collocations from textis analysed and evaluatedby applying different statistical NLP approaches ranging from just raw frequency count to the usage of more sophisticated statistical association measures. It is observed that the collocations extracted by filtering bigrams using POS taggers seem to give the best results. It is also interesting to observe that some bad collocations are selected and verified by all approaches.