sanqiang / nlp_new Goto Github PK

This repository contains code I wrote in the class Empirical Methods in Natural Language Processing.

Hopefully this can be useful to other people who are also learning NLP techniques. The code that I wrote is licensed under WTFPL version 2.

The details are described in the report.pdf files in each sub-folder. Here's a brief summary of what they are about:

1. hw1: Restore vowels to vowelless text using Finite-State Machines
2. hw2: Trigram language model with improved Kneser-Ney smoothing (best performance in class)
3. hw3: Unsupervised part-of-speech tagging using Baum–Welch Algorithm
4. hw4: Part-of-speech tagging using Conditional Random Fields (3rd best performance in class)
5. hw5: Context free grammar parsing using CKY algorithm, with vertical markovization
6. final project: a modern-Chinese-to-classical-Chinese machine translation system

Most code is written in scala (except for hw5, which is written Python: the professor kindly provided a few hundred lines of useful code in Python and I was too lazy to reimplement them, so I just learnt the language instead). You may need sbt or sbt script runner to execute them.