topobyte / osm4j Goto Github PK

View Code? Open in Web Editor NEW

44.0 4.0 7.0 2.41 MB

Library and tools for working with OpenStreetMap data and APIs

Home Page: https://jaryard.com/projects/osm4j/

Java 99.20% Shell 0.80%

java java-library osm openstreetmap osm4j

osm4j's Introduction

osm4j

This repository now contains all submodules of osm4j.

Documentation

Projects using the library

License

This library is released under the terms of the GNU Lesser General Public License.

See LGPL.md and GPL.md for details.

Download

We provide access to the artifacts via our own Maven repository:

https://mvn.topobyte.de

The packages are available at these coordinates:

de.topobyte:osm4j-core:1.3.0
de.topobyte:osm4j-geometry:1.3.0
de.topobyte:osm4j-xml:1.3.0
de.topobyte:osm4j-pbf:1.3.0
de.topobyte:osm4j-pbf-full-runtime:1.3.0
de.topobyte:osm4j-tbo:1.3.0
de.topobyte:osm4j-utils:1.3.0
de.topobyte:osm4j-extra:1.3.0
de.topobyte:osm4j-incubating:1.3.0
de.topobyte:osm4j-replication:1.3.0
de.topobyte:osm4j-testing:1.3.0

History

Formerly osm4j has been distributed among a number of independent repositories:

osm4j's People

Contributors

Stargazers

Watchers

Forkers

weexp weiga-apps mytechall dagnelies gregorywilliams thijsdejong-tomtom

osm4j's Issues

PbfWriter does not write visible-flag from metadata

The bug is in
PbfWriter:serializeMetadata(OsmEntity entity)

if you compare that with
XmlWriter:printMetadata(BuilderWriter buf, OsmMetadata metadata)

you'll see that the visible flag was forgotten.

This might also be the potential reason for:
https://lists.openstreetmap.org/pipermail/dev/2017-February/029707.html

PbfIterator fails

Hi and first of all thanks for the library. :)

I'm trying to use it from Clojure (not sure if this is relevant) and I have some issues using the PbfIterator. I can construct the object, but iterating it fails:

(-> (io/input-stream "path-to-osm.pbf")
    (PbfIterator. false) ;; works
    (iterator-seq)) ;; fails

My dependencies are as follows:

{:deps
 {de.topobyte/osm4j-xml {:mvn/version "1.2.0"}
  de.topobyte/osm4j-pbf {:mvn/version "1.2.0"}
  de.topobyte/osm4j-tbo {:mvn/version "1.2.0"}
  de.topobyte/osm4j-geometry {:mvn/version "1.2.0"}
  factual/geo {:git/url "https://github.com/Factual/geo" :sha "75722b097d559b5d073aa673fb35bfe69663b50b"}}

 :mvn/repos
 {"central" {:url "https://repo1.maven.org/maven2/"}
  "clojars" {:url "https://clojars.org/repo"}
  "topobyte" {:url "http://mvn.topobyte.de"}
  "slimjars" {:url "http://mvn.slimjars.com"}}}

This is the error message:

Unhandled java.lang.NoClassDefFoundError
   Could not initialize class
   de.topobyte.osm4j.pbf.protobuf.Fileformat$BlobHeader

              PbfUtil.java:   92  de.topobyte.osm4j.pbf.util.PbfUtil/parseHeader
              PbfUtil.java:   80  de.topobyte.osm4j.pbf.util.PbfUtil/parseHeader
          PbfIterator.java:  129  de.topobyte.osm4j.pbf.seq.PbfIterator/advanceBlock
          PbfIterator.java:  120  de.topobyte.osm4j.pbf.seq.PbfIterator/tryAdvanceBlock
          PbfIterator.java:   77  de.topobyte.osm4j.pbf.seq.PbfIterator/hasNext
                   RT.java:  517  clojure.lang.RT/chunkIteratorSeq
                  core.clj: 5729  clojure.core/iterator-seq
                  core.clj: 5721  clojure.core/iterator-seq
                      REPL:   76  wtf.nein.kosmopolit.io/eval10140
                      REPL:   74  wtf.nein.kosmopolit.io/eval10140
             Compiler.java: 7177  clojure.lang.Compiler/eval
             Compiler.java: 7132  clojure.lang.Compiler/eval
                  core.clj: 3214  clojure.core/eval
                  core.clj: 3210  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj: 1973  clojure.core/with-bindings*
                  core.clj: 1973  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  437  clojure.main/repl/read-eval-print/fn
                  main.clj:  437  clojure.main/repl/read-eval-print
                  main.clj:  458  clojure.main/repl/fn
                  main.clj:  458  clojure.main/repl
                  main.clj:  368  clojure.main/repl
               RestFn.java:  137  clojure.lang.RestFn/applyTo
                  core.clj:  665  clojure.core/apply
                  core.clj:  660  clojure.core/apply
                regrow.clj:   20  refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  202  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  201  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  834  java.lang.Thread/run

To make things more complicated I recently switched to using NixOS, so it might very well be that my setup is so unusual that you say you can't support it.

Do you have any idea however how I could debug this? I have no idea how the protocolbuffers bindings are generated or how they work. From what I've read the compiler generates java bindings, so why are there missing classes? Do I need some global packages for this?

I'd be very happy if this could somehow be made to work. Thanks in advance!

Reading changesets

Is there a way to read changes from changesets with osm4j? I searched for a way and tried to read them with EntityContainer, but it seams that there is no type available and any serialisation try failed with no generated output.

Errors parsing XML files containing user names with special characters

OSM2World users have encountered problems when working with OSM XML files containing user names with certain characters – see OSM2World issue #205 for context. These errors appear to originate from osm4j 1.2.0.

The example mentioned is user "osm-pt-account 😎". I've created a fictional example XML file for easier reproduction. (Zipped to work around GitHub's allowed file types.)

While parsing a protocol message, the input ended unexpectedly in the middle of a field.

Hi, thanks for this library.

I am trying to parse country specific pbf files I downloaded from geofabric.de and consistently get the following error when parsing any of the country files (https://download.geofabrik.de/europe/belgium-latest.osm.pbf).

The code for parsing I am using is as follows:

final PbfIterator iterator = new PbfIterator(inputStream, false);
final InMemoryMapDataSet data = MapDataSetLoader.read(iterator, false, false, true);

I get the following error when reading into MapDataSetLoader

java.lang.RuntimeException: error while reading block
	at de.topobyte.osm4j.pbf.seq.PbfIterator.ensureBeyondBounds(PbfIterator.java:228)
	at de.topobyte.osm4j.pbf.seq.PbfIterator.hasBounds(PbfIterator.java:211)
	at de.topobyte.osm4j.core.dataset.MapDataSetLoader.read(MapDataSetLoader.java:78)
	at io.referential.data.OsmFileProcessor.main(OsmFileProcessor.java:143)
Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.
	at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:107)
	at com.google.protobuf.GeneratedMessageLite.parsePartialFrom(GeneratedMessageLite.java:1584)
	at com.google.protobuf.GeneratedMessageLite.parseFrom(GeneratedMessageLite.java:1671)
	at de.topobyte.osm4j.pbf.protobuf.Fileformat$BlobHeader.parseFrom(Fileformat.java:1094)
	at de.topobyte.osm4j.pbf.util.PbfUtil.parseHeader(PbfUtil.java:92)
	at de.topobyte.osm4j.pbf.util.PbfUtil.parseHeader(PbfUtil.java:80)
	at de.topobyte.osm4j.pbf.seq.PbfIterator.advanceBlock(PbfIterator.java:129)
	at de.topobyte.osm4j.pbf.seq.PbfIterator.tryAdvanceBlock(PbfIterator.java:120)
	at de.topobyte.osm4j.pbf.seq.PbfIterator.ensureBeyondBounds(PbfIterator.java:226)
	... 3 common frames omitted

I am using osm4j-core, osm4j-pbf, osm4j-geometry 1.2.0.

Problem with NaN in PBF history files

History PBF files contain NaN coordinate values for nodes. When reading them using PBF input and writing them using PBF output, this is not stored in the same way (0 is stored instead).

Failure to read XML files that lack `lat` or `lon` values for some nodes

When converting a PBF history file to XML using osmium, the XML will contain nodes that don't have any lon or lat attributes. Attempting to parse such XML files:

Exception in thread "main" java.lang.RuntimeException: error while processing input
        at de.topobyte.osm4j.xml.dynsax.OsmXmlIterator.hasNext(OsmXmlIterator.java:138)
        at de.topobyte.osm4j.utils.executables.OsmCount.run(OsmCount.java:59)
        at de.topobyte.osm4j.utils.executables.OsmCount.main(OsmCount.java:46)
Caused by: de.topobyte.osm4j.core.access.OsmInputException: error while parsing xml data
        at de.topobyte.osm4j.xml.dynsax.OsmXmlReader.read(OsmXmlReader.java:90)
        at de.topobyte.osm4j.xml.dynsax.OsmXmlIterator$1.run(OsmXmlIterator.java:94)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NullPointerException
        at java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1838)
        at java.base/jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
        at java.base/java.lang.Double.parseDouble(Double.java:543)
        at de.topobyte.osm4j.xml.dynsax.OsmSaxHandler.emit(OsmSaxHandler.java:283)
        at de.topobyte.xml.dynsax.DynamicSaxHandler.endElement(DynamicSaxHandler.java:127)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:610)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:184)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1377)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2710)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:534)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:888)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:824)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1216)
        at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:635)
        at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:324)
        at java.xml/javax.xml.parsers.SAXParser.parse(SAXParser.java:197)
        at de.topobyte.osm4j.xml.dynsax.OsmXmlReader.read(OsmXmlReader.java:88)

TboWriter does not support visible attribute

History files do contain the visible boolean attribute which TboWriter does not store.

DynamicSaxHandler missing in xml file

Hi I am very new to this so not sure if it is maybe my fault but when I try to run the Hello World code. I get the error Exception in thread "Thread-0" java.lang.NoClassDefFoundError: de/topobyte/xml/dynsax/DynamicSaxHandler. I have downloaded the correct version of the Jar files and the xml file does not have a class named DynamicSaxHandler.

Add support for reading JOSM files

JOSM produces files with a dialect (info here: https://wiki.openstreetmap.org/wiki/JOSM_file_format)

It would be good if the XML parser was able to read those files.

Center attribute missing on OSM entities

The "center" attribute that sometime is received from an Overpass response is missing in the implementation of the osm4j parsers (especially XML parser).

The center attribute belongs to ways/relations and looks something like this: <center lat="46.2965728" lon="8.7908617"/>

Example of a whole tag (including center attribute):
<way id="458341945" version="2" timestamp="2021-04-26T18:48:09Z" changeset="103656292" uid="74900" user="FischersFritz"> <center lat="46.2965728" lon="8.7908617"/> <nd ref="4543689513"/> <nd ref="8669512158"/> <nd ref="8669512156"/> <nd ref="8669512155"/> <nd ref="8669512157"/> <nd ref="4543689514"/> <nd ref="8669512151"/> <nd ref="8669512149"/> <nd ref="8669512148"/> <nd ref="8669512150"/> <nd ref="4543689515"/> <nd ref="8669512146"/> <nd ref="8669512145"/> <nd ref="8669512144"/> <nd ref="8669512147"/> <nd ref="4543689516"/> <nd ref="8669512154"/> <nd ref="8669512153"/> <nd ref="8669512152"/> <nd ref="8669512159"/> <nd ref="4543689513"/> <tag k="building" v="yes"/> <tag k="castle_type" v="palace"/> <tag k="historic" v="castle"/> <tag k="name" v="Castello Marcacci"/> </way>