Simple, configurable, extendable tool that imports FAO data into a mongodb instance.

FAOSTAT has an existing API. This tool downloads the available bulk data dump from FAODATA (the URL is configurable) and then extracts, cleans and import data according to your specified configuration.

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

What things you need to install the software and how to install them

Python 2.7.X 
pymongo

Clone the repository as required

git clone https://github.com/keshavbahadoor/FAOSTAT-To-MongoDB

Make sure you have pymongo installed

pip install pymongo 

The data_packages.json file contains a map configuration for available data zip files, and mongodb collections for importing int. This can be specified to your requirements.

[
  {
    "filename": "Trade_LiveAnimals_E_All_Data_(Normalized)",
    "collectionname": "livestock_all"
  },
  {
    "filename": "Prices_Monthly_E_All_Data_(Normalized)",
    "collectionname": "crop_price_monthly"
  }
]

Data is imported using the command line argument import_data

python faostat_to_mongodb import_data

After the import, all downloaded and extracted files are permanently deleted so as not to consume server space.

Logs are kept on a rotation basis and can be checked for cases where data is missing or corrupted. Instances of incorrect character encoding are usually cause for issues. Log rotation can be configured in config.py.

Pull requests are welcomed :)

This project is licensed under the MIT License - see the LICENSE.md file for details