|
 |-src - main code base (python)
 |--lib - application modules
 |--sge - scripts to run jobs in parallel on a compute cluster
 |-Data - downloaded data

$ easy_install nltk beautifulsoup4 requests

$ pip install nltk beautifulsoup4 requests
```

Next execute
```
$ python -m nltk.downloader punkt averaged_perceptron_tagger
```

*** NOTE ***: you may need to setup PYTHONPATH to include the path of installed modules if those were installed into non-default locations (for instance if you installed it into your user space).

Configuration
-------------

You will need to set up the PYTHONPATH to reference the lib folder

```
# in bash
AWOIAF_ROOT=/path/to/awoiaf/
export PYTHONPATH="${PYTHONPATH}:${AWOIAF_ROOT}/src/lib"
```

Hint
----
The scrtips in the scr folder are used as the main drivers that build the data repository. You can look at those scripts as an entry point into the code. Here is a bried description for each script:

- mineCharDetails.py  - handles Ice and Fire characters' data. scripts can be used to:
  1. `python mineCharDetails -l` obtain a list of charchater names
  2. `python mineCharDetails -c "Some One"` extract data from wiki entries dedicated to character `Some One`.

- mineHousesDetails.py  - handles data related to the great houses of Westeros. scripts can be used to:
  1. `python mineHousesDetails -l` obtain a list of all the houses names mentioned in the AWOIAF wiki
  2. `python mineHousesDetails -s "House Name"`extracts data from wiki pages dedicated to `House Name`.


Look at the scripts in the scr folder to see how the modules in this app can be used and

Running multiple jobs in parallel
=================================

As this project deals with processing 1000s of wiki pages it would make sense to use parallel processing to speed things up. If you have access to a compute cluster and to the sonofgrid scheduling system (formerly called SGE) then check the folder scr/sge for scripts and documentation on how to run paralllel jobs. If you want to schedule jobs using a different system (e.g. Hadoop YARN) then you will have to figure out how to do this yourself.