encoding issue about geoip HOT 6 CLOSED

The encoding of the database file has not changed. The strings are stored internally as ISO-8859-1.

from geoip.

I think Gregory is right, which means I need to add conversions to all strings. That won't happen for a while, unless I get a solid PR from someone else.

from geoip.

Thanks for the replies and sorry for misinterpreting IRB's output.

If the db encoding is not changed than I think the problem is limited to changing https://github.com/cjheath/geoip/blob/master/lib/geoip.rb#L792, i.e. almost all fields are already in UTF-8 except for city_name (I believe that all other fields are in fact plain ASCII and city_name is the only one likely to yield characters outside of ASCII).

E.g. notice the difference between "Sao Paulo" as timezone, city_name and region_name.

geoip.city("200.220.180.2").each.map {|k| [k, k.encoding] if k.respond_to? :encoding }}
=> [["200.220.180.2", #<Encoding:UTF-8>],
 ["200.220.180.2", #<Encoding:UTF-8>],
 ["BR", #<Encoding:UTF-8>],
 ["BRA", #<Encoding:UTF-8>],
 ["Brazil", #<Encoding:UTF-8>],
 ["SA", #<Encoding:UTF-8>],
 ["27", #<Encoding:ASCII-8BIT>],
 ["S\xE3o Paulo", #<Encoding:ISO-8859-1>],
 ["", #<Encoding:ASCII-8BIT>],
 nil,
 nil,
 nil,
 nil,
 ["America/Sao_Paulo", #<Encoding:UTF-8>],
 ["Sao Paulo", #<Encoding:UTF-8>]]

(I am not sure why region_name and postal_code are ASCII-8BIT but since it's a subset of utf-8 anyway that doesn't cause issues)

So the only change would be

city.force_encoding('iso-8859-1') if city.respond_to?(:force_encoding)

to

if city.respond_to?(:force_encoding)
  city.force_encoding('iso-8859-1').encode('utf-8')
end

I can add a couple tests and make the change and send a pull request, but I couldn't find tests for the city database, are there any?

from geoip.

I just ran into this issue today as well.

The one-line fix that the previous poster suggested looks good to me:

city.force_encoding('iso-8859-1').encode('utf-8') if city.respond_to?(:force_encoding)

from geoip.

Apologies for the delay releasing this fix. I had introduced a breaking change and not had time to diagnose it. The gem is now released as 1.5.0

from geoip.

Thanks!

from geoip.