kiselev-dv / gazetteer Goto Github PK
View Code? Open in Web Editor NEWOSM ElasticSearch geocoder and addresses exporter
Home Page: http://osm.me
License: Other
OSM ElasticSearch geocoder and addresses exporter
Home Page: http://osm.me
License: Other
Good morning master,
I need your help once more, it seems that we need some tricks to resolve one of the most detailed country on OSM: Netherlands. So I ran the application as you suggested:
1st step
bzcat $inputFile | java -jar gazetteer-1.4.jar split - none
2nd step
java -jar gazetteer-1.4.jar slice --x10
3rd step
java -jar gazetteer-1.4.jar join --handlers out-gazetteer $outFile
2015-11-20 10.01.17.187 [join-stripe18544.gjson.gz] ERROR JoinSliceRunable - Join failed. File: data/stripe18544.gjson.gz.
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:2367)
...
and there also more stripes failing after these one.
Source of the file: http://download.geofabrik.de/europe/netherlands-latest.osm.bz2
What can be done here?
Thank you in advance.
adrpnt-0221010415-w148049808
Most important:
the output file is 10 bytes and join doesn't finish.
command:
java -jar gazetteer.jar split map.osm
java -jar gazetteer.jar slice
java -jar gazetteer.jar join --handlers out-gazetteer ./out.json.gz
Раз в OSM не обозначается old_name, alt_name, то хоть при геокодинге должно учитываться:
http://www.openstreetmap.org/relation/337422
Gazetteer hangs on latest OSM dumps when joining.
java -Xmx2048m -jar bin/Gazetteer.jar join --handlers out-gazetteer latest.json.gz
[join-stripe2061.gjson.gz] INFO JoinSliceRunable - stripe2061.gjson.gz done in 0:00:41.550. 57 left
Дома/города других стран оставшиеся после обрезки страны по bbox-у.
Или фильтровать по стране, или убивать адреса без страны/области.
Imports
Indexing
adrpnt-0141602584-w90640819 Калининградская область, городской округ Калининград, калининград, улица Лермонтова, 9,
adrpnt-0141716322-n1771515686 Калининградская область, городской округ Калининград, Калининград, Корсунская улица, 10,
В классе PointsReader нет проверки на присутствие атрибута timestamp. А обычно для экономии места в дампе не хранят время и автора правки.
Filter by admin boundaryes
Please add osm node ids to Gazetteer objects, currently "id" key starts with tag name and osm node id is somewhere in the middle, so is very difficult to parse.
It would be great to have structured osm id object with osm node or way id, type and tag, for ex. {'id': '25496583', 'type': 'node', 'highway': 'traffic_signals'}
or {'id': '25496583', 'type': 'way', 'highway': 'motorway'}
Nearest crossroad for poi and address.
Геокодирование по паре сотен POI внутри дома может выглядеть как overkill но для неё есть самое прямое применение в OpenLevelUp. Главное чтобы osmid в ответе приходил.
В WEB API gazetteer нужно добавить какой-нибудь параметр по которому будет ограничена область геокодинга.
Похожее было здесь: https://github.com/kiselev-dv/gazetteer/tree/develop/Gazetteer#3-how-to-filter-data-by-boundary
osm_id=w00000001
osm_id=r00000001
Очевидно, что для этого клиентам нужно знать osmid (что противоречит геокодингу без костылей), но OpenLevelUp может сделать и такое потому как данные есть от overpass.
Возможно упростить всё пользователям двумя параметрами:
restrict=admin_level + restrict_query=Москва
restrict=building + restrict_query=Афимолл Сити
restrict=mall + restrict_query=Centre Commercial Le Coudoulet
Самая простая реализация это добавить метки is_in_Афимолл_Сити всем объектам внутри здания.
Моллы/retail здания можно выделить из всех остальных "домов" для уменьшения индекса.
Для админ границ не такая востребованная фича, может подождать.
Матчить 2 параметра по этим меткам да хоть на равенство (уже лучше Номинатима).
Можно сортировать одинаковые результаты по дистанции к lat/lot/zoom параметоров из клиентского приложения (у OpenLevelUp в URL есть они). Т.е. перезаписывая веса внутренние веса ES на метры до lat/lot.
Можно эту пост-сортировку до центра экрана сделать опциональной.
PS.
тикет в OpenLevelUp https://framagit.org/PanierAvide/OpenLevelUp/issues/10
тикет в Photon komoot/photon#226
пример , Гусевский район, Гусевское городское поселение, Гусев, улица
вариант 100% попаданий не вижу, варианты:
не забыть про Питер и Москву
Adding a GUI for gazetter will improve usability, specialy when exporting.
Also adding an option for export sql or sqlite will make it easy and faster when using on android.
Иначе не находит совсем https://github.com/search?utf8=%E2%9C%93&q=osm+geocoder&type=Repositories
Вот этот находит: https://github.com/chatman/osm-geocoder
Hi,
while working with the gazetteer (btw. great tool!) to extract addresses from OSM I found in the Readme the note, that it is possible to output also to CSV-format. Unfortunately I was not able to figure out how to do this. Could you give me please a little example?
Thanks a lot!
subj
Add different fuzzy matchers for street names and places names matching.
Must be accesible from full text addresses formatters and so on.
Can you make data extracts for Romania also, please?
Mend common types of geometry errors:
по крайней мере актуально для бывш. СССР
Hi,
in data extracts produced by Gazetteer objects don't follow GeoJson object naming because are all-lowercase.
Could you please fix that as GeoJSON specification requires:
A geometry is a GeoJSON object where the type member's value is one of the following strings: "Point", "MultiPoint", "LineString", "MultiLineString", "Polygon", "MultiPolygon", or "GeometryCollection".
Another question I have: Is it possible to extract only addresses, and no highways or other data?
Thanks a lot!
Add csv export.
Use config string or groovy script.
Add addrN full support.
Resource: Striper/src/main/java/me/osm/gazetter/addresses/AddressesParser.java
Location: line 285
Add template string or maybe file with js or groovy formatter script.
Address is the simple json object so using js or groovy with json.org lib preimported should be simple enough.
Российская Федерация, Российская Федерация, Северо-Западный федеральный округ,
mvn clean compile assembly:single -f Gazetteer/pom.xml
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Gazetteer 1.4-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[WARNING] The POM for me.osm.osm-doc:osm-doc-java:jar:0.11 is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.868s
[INFO] Finished at: Sat Sep 19 02:34:41 CEST 2015
[INFO] Final Memory: 6M/236M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project Gazetteer: Could not resolve dependencies for project me.osm.Gazetteer:Gazetteer:jar:1.4-SNAPSHOT: Failure to find me.osm.osm-doc:osm-doc-java:jar:0.11 in http://raw.githubusercontent.com/kiselev-dv/mvn-repository/master/releases/ was cached in the local repository, resolution will not be reattempted until the update interval of osm-doc-mvn-repo has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
Optimize point location
Add points filtration by Hilbert hash prefix
Resource: Striper/src/main/java/me/osm/gazetter/pointlocation/PLTask.java
Location: line 101
http://www.openstreetmap.org/way/230881424
Ищем " Ульяновск, 1-й Инзенский переулок, 20/31 " - находит "1-й Инзенский переулок, 20"
Совсем не ищет "Ульяновск, 3-й Инзенский переулок, 21"
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.