idlesign / pysyge Goto Github PK

View Code? Open in Web Editor NEW

26.0 6.0 11.0 67.28 MB

API to access data from Sypex Geo IP database files from your Python code

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

python geoip geo sypexgeo

pysyge's Introduction

pysyge

http://github.com/idlesign/pysyge

What's that

pysyge is an API to access data from Sypex Geo IP database files from your Python code.

For more information about Sypex Geo databases and their features please visit http://sypexgeo.net.

Direct DB link that might became broken over time:

Sypex Geo City DB - http://sypexgeo.net/files/SxGeoCity_utf8.zip

Requirements

Python 3.6+

Attention

This version of pysyge works with Sypex Geo DB version 2.2 or above.

The structure of a dictionary returned by GeoLocator.get_location() was preserved in a backward compatible manner as much as possible, yet it's advised to update your code to use data from info sub dictionary.

This version of pysyge works with UTF-8 Sypex Geo Databases. Texts returned by pysyge are UTF-8.

Quickstart

Download Geo IP database file from http://sypexgeo.net/ (example below uses SxGeoCityMax.dat file).

Application sample

# Import all we need from pysyge module.
from pysyge import GeoLocator, MODE_BATCH, MODE_MEMORY

# Create GeoLocator object to access API
# from 'SxGeoCityMax.dat' using fast memory mode.
geodata = GeoLocator('SxGeoCityMax.dat', MODE_BATCH | MODE_MEMORY)

# Let's get some meta information.
print('DB version %s (%s)' % (geodata.get_db_version(), geodata.get_db_date()))

# Request geo information for 77.88.21.3 IP address.
# Getting detailed information, including region info.
location = geodata.get_location('77.88.21.3', detailed=True)

# Print out some lyrics.
# Most interesting data is under `info` in `city`, `country` and `region` dictionaries
print('%s (%s) calling. All the circuits are busy.' % (
    location['info']['city']['name_en'], location['info']['country']['iso']))

pysyge's People

Contributors

Stargazers

Watchers

Forkers

method163 psaria pekopt mdlx mishin nikedadi polozhevets illuzzion yakovlevvs unixlamaster unclestranger

pysyge's Issues

А как можно получить список регионов России к примеру?

Getting a ValueError if called get_location with `detailed=True` argument on latest SxGeoCity.dat (2013.04.22)

from pysyge.pysyge import GeoLocator, MODE_BATCH, MODE_MEMORY
ip = '95.140.92.49'
db = GeoLocator('SxGeoCity.dat', MODE_BATCH | MODE_MEMORY)
# Does stuff
db.get_location(ip)
{'city': '...',}

# Fails
db.get_location(ip, detailed=True)
"""
local/lib/python2.7/site-packages/pysyge/pysyge.pyc in get_location(self, ip, detailed)
    290             location = self._parse_location(seek)
    291             if detailed:
--> 292                 return self._parse_location_details(location)
    293             return location
    294         else:

local/lib/python2.7/site-packages/pysyge/pysyge.pyc in _parse_location_details(self, city)
    253                 region = self._fh.read(self._max_region).split('\0')
    254             city['region'] = region[0]
--> 255             city['tz'] = self._tz[int(region[1])]
    256         else:
    257             city['region'] = ''

ValueError: invalid literal for int() with base 10: ''

"""

When city is not determined, answer contans 'True' instead of name

Hi.
When city is not determined from database (sypex online tester returns only country) I get True instead of city and region names:
http://prntscr.com/q73qb8

geo_obj.get_location('91.210.107.80', detailed=True)

{"country_id": 185, "country_iso": "RU", "region_id": 0, "city": true, "lon": 100.0, "lat": 60.0, "fips": "0", "info": {"city": {"id": 0, "lat": 60.0, "lon": 100.0, "name_ru": true, "name_en": true}, "region": {"id": 0, "name_ru": true, "name_en": true, "iso": true}, "country": {"id": 185, "iso": "RU", "lat": 60.0, "lon": 100.0, "name_ru": "Россия", "name_en": "Russia"}}, "region": true, "tz": ""}

This algorithm is incorrect

pysyge/pysyge/pysyge.py

Line 186 in 3d57522

while (max_ - min_) > 8:

For example, ipn == str_[start:start + 3], you should increase min_ border, because 'str_[start:start + 3]' is the left border of a new range.

generates an exception when an IP address is not in the database

generates an exception like

TypeError: 'bool' object has no attribute '__ getitem __'

For example: type into https://sypexgeo.net/ru/demo/ list of IP's

163.53.186.51
185.26.183.14
185.26.183.14
45.64.99.151
104.235.29.75
104.236.81.126
104.41.151.86
154.72.48.97
45.63.53.242
104.238.83.28
103.230.136.250

solution:

instead

location = geodata.get_location(proxy.strip().split(':')[0], detailed=False)
iso = location['country_iso']

use

try:
    location = geodata.get_location(proxy.strip().split(':')[0], detailed=False)
    iso = location['country_iso']
except:
    iso = 'UNKNOWN'
    pass

Incorrect read

24.231.175.176 - error in 2020.07.31 building

MODE_BATCH incorrect city detection

Incorrect detection for some IPs. 109.252.119.71 as an example
PHP API:

docker run --rm -v $(pwd):/app/ php:7-cli php /app/sxgeo_sample.php

IP: 109.252.119.71
array (
  'city' => 
  array (
    'id' => 524901,
    'lat' => 55.75222,
    'lon' => 37.61556,
    'name_ru' => 'Москва',
    'name_en' => 'Moscow',
  ),
  'region' => 
  array (
    'id' => 524894,
    'name_ru' => 'Москва',
    'name_en' => 'Moskva',
    'iso' => 'RU-MOW',
  ),
  'country' => 
  array (
    'id' => 185,
    'iso' => 'RU',
    'lat' => 60,
    'lon' => 100,
    'name_ru' => 'Россия',
    'name_en' => 'Russia',
  ),
)array (
  'city' => 
  array (
    'id' => 524901,
    'lat' => 55.75222,
    'lon' => 37.61556,
    'name_ru' => 'Москва',
    'name_en' => 'Moscow',
  ),
  'country' => 
  array (
    'id' => 185,
    'iso' => 'RU',
  ),
)array (
  'Created' => '2023.11.16',
  'Timestamp' => 1700171947,
  'Charset' => 'utf-8',
  'Type' => 'SxGeo City EN',
  'Byte Index' => 224,
  'Main Index' => 1775,
  'Blocks In Index Item' => 3376,
  'IP Blocks' => 5994260,
  'Block Size' => 6,
  'City' => 
  array (
    'Max Length' => 127,
    'Total Size' => 2682917,
  ),
  'Region' => 
  array (
    'Max Length' => 175,
    'Total Size' => 109379,
  ),
  'Country' => 
  array (
    'Max Length' => 147,
    'Total Size' => 9387,
  ),
)`

PYSYGE

from pysyge.pysyge import GeoLocator, MODE_BATCH, MODE_MEMORY
geodata = GeoLocator('/tmp/SxGeoCity.dat', MODE_BATCH | MODE_MEMORY)
print('DB version %s (%s)' % (geodata.get_db_version(), geodata.get_db_date()))
geodata.get_location('109.252.119.71', detailed=True)

DB version 22 (2023-11-17 04:59:07)
Out[3]: 
{'country_id': 174,
 'country_iso': 'PL',
 'region_id': 3337499,
 'city': 'Щецин',
 'lon': 14.55302,
 'lat': 53.42894,
 'fips': '0',
 'info': {'city': {'id': 3083829,
   'lat': 53.42894,
   'lon': 14.55302,
   'name_ru': 'Щецин',
   'name_en': 'Szczecin'},
  'region': {'id': 3337499,
   'name_ru': 'Западно-Поморское воеводство',
   'name_en': 'Zachodniopomorskie',
   'iso': 'PL-ZP'},
  'country': {'id': 174,
   'iso': 'PL',
   'lat': 52.0,
   'lon': 20.0,
   'name_ru': 'Польша',
   'name_en': 'Poland'}},
 'region': 'Западно-Поморское воеводство',
 'tz': ''}

Compatible with Sypex Geo 2.2.1

Database was updated and now lib throw exception.

Exception Type: IndexError
Exception Value: list index out of range

Traceback:

.../application/core/libs/pysyge.py in get_location
296: seek = self._get_pos(ip) ...

.../application/core/libs/pysyge.py in _get_pos
219: part = self._search_idx(ipn, int(floor(blocks['min'] / self._range)), int(floor(blocks['max'] / self._range) - 1)) ...

.../application/core/libs/pysyge.py in _search_idx
152: if ipn > self._m_idx_set[offset]: ...

get all cities?

Thx for this lib.
Can u help me, i'm need to get all cities from db.

Periodically, I get an error on the site

Hello.
I can't understand - an error occurs periodically, help me figure it out.

  File "/var/www/.venvs/python38/lib/python3.8/site-packages/pysyge/pysyge.py", line 446, in get_location
    return self._parse_location(seek, detailed=detailed)
  File "/var/www/.venvs/python38/lib/python3.8/site-packages/pysyge/pysyge.py", line 308, in _parse_location
    city = self._read_data_chunk(self._TYPE_CITY, start_pos, self._max_city)
  File "/var/www/.venvs/python38/lib/python3.8/site-packages/pysyge/pysyge.py", line 291, in _read_data_chunk
    return self._parse_pack(self._pack[data_type], raw)
  File "/var/www/.venvs/python38/lib/python3.8/site-packages/pysyge/pysyge.py", line 417, in _parse_pack
    val_real = val_real.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 0: invalid start byte

What can be and where to look for details? Thanks!)

Exception while trying to parse ip

Hi.
I get exception, trying to parse address 49.206.213.75.
Environment:

latest Sypex version (2019.11.19)
library version (1.0.0)
Python 3.7 virtualenv

How to reproduce:

from pysyge.pysyge import GeoLocator, MODE_MEMORY
loc = GeoLocator(GEOAPI_DB_PATH, MODE_MEMORY)
loc.get_location('49.206.213.75', detailed=True)

Trace:

  File "pysyge/pysyge.py", line 396, in get_location
    seek = self._get_pos(ip)
  File "pysyge/pysyge.py", line 230, in _get_pos
    return self._search_db(self._db, ipn, min_, max_)
  File "pysyge/pysyge.py", line 183, in _search_db
    return int(str_[start:start + 3].encode('hex'), 16)

Suggestion:
Change str_[start:start + 3].encode('hex') to str_[start:start + 3].hex()
Docs: https://docs.python.org/3/library/stdtypes.html#bytes.hex

Неправильно определяются города

Для IP адресов 159.255.81.209 и 159.255.82.245 (оба Иркутск, сотовый оператор Байкалвестком) выдаются соотв. Амстердам и Тоскана. При этом глюк имеет место быть только при использовании pysyge. Сам сайт sypexgeo.net все показывает корректно.