hsldevcom / digitransit-geocoder Goto Github PK

Debian has a custom version of pip, which broke in testing: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786580

see #1

Search for "sampsantie", you will see that city is "helsinki". Should be "Helsinki"

ERROR:tornado.access:500 GET /reverse/60.19,24.93 (172.17.2.247) 20003.41ms

Caused by: java.lang.OutOfMemoryError: Java heap space
[2015-04-13 13:10:04,943][DEBUG][action.search.type ] [White Pilgrim] [reittiopas][4], node[lH7ZBJ4yTV2SBnoO7XGXHQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@29fcc13c] lastShard [true]

http://www.elastic.co/guide/en/elasticsearch/guide/current/_important_configuration_changes.html needs to be read carefully when setting up the real thing.

For example http://matka.hsl.fi/geocoder/search/Helsinki/Helsingin%20rautatieasema

Why such address exists?
http://matka.hsl.fi/geocoder/search/helsinki/kivikallio

Example: Virsutie 5 finds Virsutie 4-6.

• Decide whether we use pelias API or ElasticSearch API
• Search metadata parameters support (See: #1)

Now these return response in different format:

• http://matka.hsl.fi/geocoder/search/helsinki/mannerheimintie
• http://matka.hsl.fi/geocoder/search/helsinki/mannerheimintie/77

Unify response.

All fields in "/search/address" should be included to "/search/address/number"

Document how implementations works see #1

• What are endpoints
• How they should be used
• What do endpoints return
• How do endpoints vary

Let's start by writing documentation to git repo readme.

Results are ordered as described in #1

As a quick fix to get UI more responsive to search, reduce amount of returned results to 10 per returned type.

Now there is problem that UI basically jams when trying to parse too many results.

ERROR:tornado.application:Uncaught exception GET /interpolate/mannerheimintie/10 (::1)
HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/interpolate/mannerheimintie/10', version='HTTP/1.1', remote_ip='::1', headers={'User-Agent': 'curl/7.38.0', 'Host': 'localhost:8888', 'Accept': '/'})
Traceback (most recent call last):
File "/home/hukka/addressimport/lib/python3.4/site-packages/tornado/web.py", line 1302, in _stack_context_handle_exception
raise_exc_info((type, value, traceback))
File "", line 3, in raise_exc_info
File "/home/hukka/addressimport/lib/python3.4/site-packages/tornado/stack_context.py", line 314, in wrapped
ret = fn(_args, *kwargs)
File "./app.py", line 34, in on_response
self.write(self.transform_es(json.loads(response.body.decode('utf-8'))))
File "./app.py", line 164, in transform_es
fraction = (self.streetnumber - int(street["min" + self.side])) /
TypeError: int() argument must be a string or a number, not 'list'
ERROR:tornado.access:500 GET /interpolate/mannerheimintie/10 (::1) 33.33ms

Check that languages work as described in #1

Either elastic-wait needs to be modified to check shard status, or app.py needs to handle 503 gracefully.

We should get from OSM:

• addresses
• "building characters" ("talokirjaimet")
• Stairways
• places

e.g.
Opastinsilta 1
Opastinsilta 2
Opastinsilta 3
...
Opastinsilta 10

Start by doing some user stories about the actual usage. After this, we will decide how to proceed with geocoding.

See #1

If we can't get accurate address information for the whole country, we need to use the road/street data from National Land Survey with interpolation.

Often OSM has data in only one language, and does not tell what that language is.

• Setup ES
• OSM data
• seudullinen osoiteluettelo

See #1

Currently GTFS has only Finnish

For example Opastinsilta 6a is shared with HSL and local pub. Return both or just a centroid. Currently geocoder returns one 6a and one 6A each...

Pelias demo has problems with data import, so we need a customized version to test the existing features

• Switch address numbering from US to Finnish style
• Import all address data also in Swedish to test multilingual search

• Reduce amount of results that geocoder returns to 20
• Ensure that API returns needed data for UI (See HSLdevcom/digitransit-ui#67)

Imported Datasources

• Seudullinen osoiteluettelo
• MML kunnat
• Palvelukartta pääkaupunkiseutu
• OSM
• Valtakunnallinen palvelukartta
• Liikuntapaikat (lipas-service)

Searchable data

• Geocoding (position for given text):
  • Address
  • Stop short number
  • Stop long number
  • POI
  • District (kaupunginosa)
  • City
• Reverse geocoding (text for given position)
  • Address

Search metadata parameters

• Search modes
  • "Pääkaupunkiseutu tagged": Only search for tagged data for "pääkaupunkiseutu".Note: "Kaupunkiseutu" is not an area on map. It is all data that is tagged for it.
  • "All": Search in all of Finland
• Zoom level
  • Geocoding should take zoom level as parameter. This should affect whether addresses are returned. I.e. on low zoom level only city district or city should be returned.
• True/False whether services (e.g. libraries) should be returned. Check http://dev.hel.fi/servicemap
• How many search results are returned per "result type" (address, poi, ...)

Matching of search results

• Match should be full text search. E.g. "man" should match "man"
• Match should work with typos. Consider Levenshtein algorithm.
• Matching should work with "typos" and "suggestions"
• Address search should recognize match without comma before city in search string e.g: "opastinsilta 6 helsinki"
• Address search should recognize match without the name of the city e.g: "linnankoskenkatu 10" finds: "Linnankoskenkatu 10, Helsinki".

Language support

• Support: Finnish and Swedish.
• If UI language is in Swedish: Return results in Swedish
• If UI language is in Any other language: Return results in Finnish

Returning results

• No duplicates should be returned if same target is found from multiple datasources.

Ordering of results

1. Address
2. POI (e.g. Finlandia house, Tampere)
3. Service (e.g. Library)
4. Stop name
5. Stop number
   If same street name exists in multiple times, order is according alphabetic order of city name. e.g. "koivutie 4" gives first "Koivutie 4, Kirkkonummi".

Search speed

• Search should be as fast as possible. One second is too slow, it should be around 100 - 300ms.

Documentation

• Document how geocoding is implemented

Requirements on UI

• When user clicks on search box, system should immediately show:
  1. Favourites. Order these by distance from current location if possible.
  2. Previous searches

Integration tests for each API endpoint, varying capitalization of words, including non latin chars

Currently only capital area services are imported