Code Monkey home page Code Monkey logo

geodis's Introduction

Build Status

Geodis - a Redis based geo resolving library


Geodis is a simple and fast python module that allows you to convert IP addresses and latitude/longitude coordinates into geographical locations such as cities, zipcodes and countries.

It currently supports cities worldwide, and zipcode areas in the US (of course each of these includes higher level data such as country). But it is written in an extensible way, so that adding a new layer of objects and indexing them is very simple.

Geodis is fast, since it uses redis, which is a very fast in memory database, and geohashing to index coordinates.

A single thread, single process python program can resolve about 2000 ips and 3000 lat/lon pairs per second on a regular desktop machine, when the database is fully loaded with IP ranges, zipcodes and all major cities in the world.

Geodis - Getting started

Before you jump into python shell to test geodis , you must first ensure that redis server is running and on which port. If you do not have a clue how to get started with redis-server here is a link to get started, download install.

once redis-server is up and running you can test in python shell as shown below:

>>> import redis
>>> rs = redis.Redis("localhost")
>>> rs
Redis<ConnectionPool<Connection<host=localhost,port=6379,db=0>>>

The above line showing host and port ensures that redis server is running and accepting incoming connections at port 6379.

USAGE

>>> import redis
>>> import geodis.city
>>> conn = redis.Redis()

#getting a city by lat,lon
>>> print geodis.city.City.getByLatLon(31.78,35.21, conn)
Location: {'name': 'West Jerusalem', 'country': 'Israel', 'lon': '35.21961', 'zipcode': '', 'state': 'Jerusalem District', 'lat': '31.78199'}

#getting a location by ip
>>> print geodis.iprange.IPRange.getCity('62.219.0.221', '62.219.0.221', 31.78, 35.21, conn)
Location: {'name': 'West Jerusalem', 'country': 'Israel', 'lon': '35.21961', 'zipcode': '', 'state': 'Jerusalem District', 'lat': '31.78199'}

Geodis can also be used as a command line utility

$ geodis -P  188.127.241.156 -p 6379
Location: {'name': 'Crosby', 'country': 'United Kingdom', 'lon': '-3.03333', 'zipcode': '', 'state': 'England', 'key': 'loc:crosby:united kingdom:england:', 'lat': '53.47778'}

$ geodis -L  40.90732,-74.07514 -p 6379
Location: {'name': 'Rochelle Park', 'country': 'United States', 'lon': '-74.07514', 'zipcode': '', 'state': 'New Jersey', 'key': 'loc:rochelle park:united states:new jersey:', 'lat': '40.90732'}

IMPORTING DATA

Geodis needs to import its data into redis. In the data folder you will find a list of all cities in the world, and a zipcode database.

The data files should be where the geodis files are installed if you've installed from pip (e.g /usr/local/lib/python2.7/site-packages/geodis/data/cities1000.json), or in the source tree if you've cloned this repo.

data is imported using a utility called geodis.py. run ./geodis.py --help for more details on importing it.

Examples:

  • Cities are imported by running

      geodis -g -f <data directory>/cities1000.json -p 6379
    
  • Zipcodes are imported by running

      geodis -z -f <data directory>/zipcode.csv -p 6379
    

** IMPORTANT: IP to location data is not provided, you need to buy an ip resolving database that can resolve ip ranges to lat,lon pairs **

Refreshing countries mapping:

The data is already generated but if you ever need to update, use:

    python external/geonames/update.py > geodis/countries.py

INSTALLING:

  • pip install geodis

RUNNING:

  1. Install geodis

  2. Install redis

  3. Import data as described above.

geodis's People

Contributors

dvirsky avatar lavie avatar sanfx avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

geodis's Issues

Some extra user friendliness

Hi,

I think this is a great package that should be known more. Googles API allows only a few requests per day for free and has a network overhead (although it seems to offer more information).

For more than a handfull of requests I found Geodis a nice alternative, had to work a bit with the redis API (figuring out the correct db data were stored in) and with setting up symlinks to import the package.

Also the geobox branch is not switched to master yet and maybe that is because it threw out an error at me when I attempted to store data to redis with it. I figure its just a configuration error on my part.

So, to sum up, I guess this is not that much of an issue, as just thanks and maybe to lower the entry barrier a little bit more for beginners (althought API felt quite intuitive so far).

Thanks for sharing.

Need a way to only get zip code if within the US

Seems like the geodis.ZIPCode.getByLatLon function will return a zip code and the country 'United States' no matter where in the world the latitude/longitude points are. Is there an easy way to determine country for these?

I assumed I could do geodis.City.getByLatLon first, but I cannot seem to get it to work reliably. Perhaps I am not getting the geonames data file loaded correctly or something. Here is what I see:

I call:

    broadlocation = geodis.City.getByLatLon(latitude,longitude, conn)

and it fails fairly often after a few lookups (which the ZIPCode.getByLatLon never fails upon). This is what the failure looks like:

Traceback (most recent call last):
File "test.py", line 22, in
broadlocation = geodis.City.getByLatLon(latitude,longitude, conn)
File "/Users/jasontitus/Documents/geo-data/doat-geodis-d56cd38/src/location.py", line 98, in getByLatLon
return cls.getByGeohash(geoKey, redisConn)
File "/Users/jasontitus/Documents/geo-data/doat-geodis-d56cd38/src/location.py", line 157, in getByGeohash
return cls.load(str(candidates[selected][0]), redisConn)
File "/Users/jasontitus/Documents/geo-data/doat-geodis-d56cd38/src/location.py", line 84, in load
d = redisConn.hgetall(str(key))
File "/Library/Python/2.6/site-packages/redis-2.4.1-py2.6.egg/redis/client.py", line 978, in hgetall
File "/Library/Python/2.6/site-packages/redis-2.4.1-py2.6.egg/redis/client.py", line 239, in execute_command
File "/Library/Python/2.6/site-packages/redis-2.4.1-py2.6.egg/redis/client.py", line 245, in parse_response
File "/Library/Python/2.6/site-packages/redis-2.4.1-py2.6.egg/redis/connection.py", line 207, in read_response
File "/Library/Python/2.6/site-packages/redis-2.4.1-py2.6.egg/redis/connection.py", line 67, in read_response
File "/Library/Python/2.6/site-packages/redis-2.4.1-py2.6.egg/redis/connection.py", line 36, in read_response
redis.exceptions.ConnectionError: Socket closed on remote end

Any idea what the issue might be?

I found an error when importing data. How should I do?

File "/usr/local/lib/python2.7/dist-packages/geodis-2.0.3-py2.7.egg/geodis/city.py", line 57, in init
self.country = countries.get( kwargs.get('country', None), kwargs.get('country', '')).strip()
AttributeError: 'list' object has no attribute 'get'
Could not import geonames database...

Couldn't install

Hi,

I couldn't install geodis using PIP. There is an error: Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?

Could not import zipcodes data

I am trying to import the zipcodes data into the redis instance as specified on the ReadMe.
Using the command ./geodis -z -f data/zipcodes.csv -p 6369. All imports failed with a similar error to the one listed below. I have had no problem importing the cities.json file however.
KeyError: 'alaska'
2018-10-17 19:13:01,291 ERROR in zipcodes.runImport (zipcodes.py:129): Could not import line #43191: Ketchikan, {'lat': 61.385, 'code': 'AK', 'lon': -152.2683, 'name': 'Alaska'}: 'alaska'
Traceback (most recent call last):
File "/home/u1604342/.local/lib/python2.7/site-packages/geodis/provider/zipcodes.py", line 108, in runImport
stateId = features[stateName.lower()]['id']

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.