wdtinc / mapbox-vector-tile-java Goto Github PK

I created a failing test against some test vector tiles from the mapbox/vector-tile-js repo.

See: testMultiPolygon() in snodnipper@fc38710

May I ask if this is a known limitation?

Found an issue while rendering the osm roads layer. Some of the line geometries does not render properly at lower zoom levels specially where the line segments form a ring.

Original geometry

MVT Geometry

Original geometry WKT
MULTILINESTRING ((545022.4644456336 6867178.738599846, 545023.7891475739 6867185.429774573, 545024.813286887 6867190.64415437, 545029.8894556686 6867190.206583908, 545036.2791944387 6867192.339740138, 545040.0751890754 6867195.7309127385, 545042.7913846518 6867201.255245361, 545043.1476070203 6867206.34201043, 545042.4574261782 6867209.477938888, 545040.03066128 6867213.944815833, 545037.4591810413 6867216.4608537285, 545032.939609716 6867218.794570745, 545028.5536217794 6867219.469160945, 545023.5331127449 6867218.5757847475, 545020.4606947971 6867217.026050759, 545015.5960330492 6867211.611100222, 545014.0486921292 6867206.761349631, 545014.1488796688 6867202.185083897, 545016.5088528739 6867196.4966615895, 545020.0822085281 6867192.868471254, 545024.813286887 6867190.64415437))

I'd tried to create mvt from polygon but there is a problem in rendering it.
I Read.me explain something about Buffering Polygons but I can't find this method:

// Build buffered MVT tile geometry
final TileGeomResult bufferedTileGeom = JtsAdapter.createTileGeom(
        JtsAdapter.flatFeatureList(inputGeom),
        tileEnvelope, clipEnvelope, geomFactory,
        DEFAULT_MVT_PARAMS, ACCEPT_ALL_FILTER);

It seems that rings orientation in polygons is wrong. The specs says that exterior ring must be clockwise in screen coordinates. But orientation is flipped when JTS geometry is encoded to mvt, so preparation step should do the opposite orientation when it operates with JTS geometry. Mapbox encoder implementation does it like I say https://github.com/mapbox/geojson-vt/blob/d3bf094b1329dd84dcc8446ba407c9830469b808/src/tile.js#L113

Because of this issue extruded polygons are displayed inside out in mapboxgl viewer

When I swapped orientation checks for exterior and interior rings in the following file then 3d polygons became normal

Equality is failing because JTS does not consider user objects (attributes).

    @Test
    public void testEqualityWithUserAttributes() {
        // two identical points
        Point point = createPoint(new int[]{50, 0});
        Point point2 = createPoint(new int[]{50, 0});

        // noise
        Point point3 = createPoint(new int[]{60, 10});

        // add attributes only to the first point
        Map<String, Object> attributes = new LinkedHashMap<>();
        attributes.put("id", "test");
        point.setUserData(attributes);

        // define two layers (geometries are the same _but_ the attributes are not)
        JtsLayer layer = new JtsLayer("Point", Arrays.asList(point, point3));
        JtsLayer layer2 = new JtsLayer("Point", Arrays.asList(point2, point3));

        assertFalse(layer.equals(layer2));
    }

My suggestion is that we create an MvtFeature class that encapsulates both the geometry and the attributes (e.g. forcing underlying long values rather than allowing ints). We can then compare the JTS geometries and attributes independently. Such a class would also allow us to remove the committed but dirty code from the JtsLayer.

Furthermore, it would be good to consider the role of the JTS classes. Specifically, we marked .getGeometries as read-only. I committed code with Collections.unmodifiableList(...) whilst working on the equality aspects. Clearly we could keep a mutable list and add, say, an addFeature method...but then we'd lose the immutable benefits (final/unmodifiable).

I wanted to get some visibility of this early before too much effort goes into it.

Note: there is an open issue on equality with JTS.

Draft code fixes: https://github.com/OrdnanceSurvey/mapbox-vector-tile-java/tree/bug/missingattributes

I have spent the day running code quality tools to get this going on old versions of Android. I can feedback as a separate issue - I needed Java 1.7 API compatibility, for instance.

Anyhow, these lines fall through and look suspicious.

https://github.com/wdtinc/mapbox-vector-tile-java/blob/master/src/main/java/com/wdtinc/mapbox_vector_tile/adapt/jts/MvtReader.java#L535

https://github.com/wdtinc/mapbox-vector-tile-java/blob/master/src/main/java/com/wdtinc/mapbox_vector_tile/adapt/jts/MvtReader.java#L592

Wrong feature count used when flattening GeometryCollection in JTSAdapter#flatFeatureList(Geometry).

I am not exactly sure why this happens, but when I try to decode tiles from ThunderForest, I receive an Exception:

com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either that the input has been truncated or that an embedded message misreported its own length.
        at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:70)
        at com.google.protobuf.CodedInputStream.readRawBytesSlowPath(CodedInputStream.java:1211)
        at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:517)
        at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:516)
        at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:493)
        at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:611)
        at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:281)
        at com.google.protobuf.CodedInputStream.readGroup(CodedInputStream.java:424)
        at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:520)
        at com.google.protobuf.MessageReflection.mergeFieldFrom(MessageReflection.java:757)
        at com.google.protobuf.GeneratedMessage$ExtendableMessage.parseUnknownField(GeneratedMessage.java:847)
        at com.wdtinc.mapbox_vector_tile.VectorTile$Tile.<init>(VectorTile.java:77)
        at com.wdtinc.mapbox_vector_tile.VectorTile$Tile.<init>(VectorTile.java:43)
        at com.wdtinc.mapbox_vector_tile.VectorTile$Tile$1.parsePartialFrom(VectorTile.java:4802)
        at com.wdtinc.mapbox_vector_tile.VectorTile$Tile$1.parsePartialFrom(VectorTile.java:4796)
        at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:192)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:210)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:215)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
        at com.wdtinc.mapbox_vector_tile.VectorTile$Tile.parseFrom(VectorTile.java:4341)
        at com.wdtinc.mapbox_vector_tile.adapt.jts.MvtReader.loadMvt(MvtReader.java:120)

I haven't debugged this in greater detail. So I don't know if this an issue from Thunderforest or from this library. Thunderforest specifies that they encode in the MVT schema. But I hadn't had any problems decoding tiles from other providers so far.

Since probably not everyone wants to sign up for a key to try this, I attached one tile from ThunderForest in this issue, downloaded from: https://tile.thunderforest.com/thunderforest.transport-v1.json?apikey={apikey}
thunderforest.vector.pbf.zip

From working with the library, the TagKeyValueMapConverter does not preserve the ordering of attributes.

It would be great if we could use, say, a LinkedHashMap so that attributes look consistent.

For example, geo analysts like to see:

id: 123456789
name: Paul
age: 47

without the order of the above being arbitrary.

For tests I created a TagKeyValueMapConverterOrder class but perhaps the overhead isn't so bad for default use?

I just started test your library and saw there seems to be an issue with parsing Polygons.

When I try to read a polygon the outer and inner rings seem to be switched and many polygons are not accepted because the library thinks that the polygon has no outer ring. The problem seems to be created in MvtReader#classifyRings when running CGAlgorithms.signedArea(r.getCoordinates()).

For example have a look at this tile. The first polygon is a water polygon that has no inner, just an outer. Same thing happens for Mapzen mvt, they also provide a nice geojson debugging version of the tile, where you can see that it's just a big simple polygon.

Change locationtech JTS core to 1.15.1 for now to preserve compatibility for Android projects. Still an upgrade from 1.15.0.

JTS 1.17.0 changed the signature of the Polygon.getExteriorRing() method and it now returns a LinearRing instead of a LineString. This leads to this error when you try to use a current version of this library with JTS 1.17.0:

java.lang.NoSuchMethodError: org.locationtech.jts.geom.Polygon.getExteriorRing()Lorg/locationtech/jts/geom/LineString;

at com.wdtinc.mapbox_vector_tile.adapt.jts.JtsAdapter.toFeature(JtsAdapter.java:357)
at com.wdtinc.mapbox_vector_tile.adapt.jts.JtsAdapter.toFeatures(JtsAdapter.java:300)
at com.wdtinc.mapbox_vector_tile.adapt.jts.JtsAdapter$toFeatures$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:157)

The upgrade is pretty easy. There is only one code change in a test (in MvtReaderTest PackedCoordinateSequenceFactory constructor now only takes one parameter not two). I will follow up with a PR.

Thanks!
Jared

please add documentation to add vector to mapbox map android

Hi all,

I noticed I'm having issues displaying polygons crossing Antimeridian. They usually either display totally broken or don't display at all. Is there any specific way to deal with those polygons?

Hi! I am working on building a simple MVT service to be used by Mapbox.gl but am a little confused with the createTileGeom function signature.

I have a service endpoint...

http://localhost/tiles/{z}/{x}/{y}

Both the Envelope and MvtLayerParams appear to take in x/y or an extent. How might I go about only displaying tiles that are within the users viewable extent with only the z/x/y are sent to the server?

This's one of my data -----POLYGON ((116.378477 39.960438, 116.378312 39.960438, 116.378312 39.960344, 116.378371 39.960344, 116.378365 39.960303, 116.378347 39.960261, 116.378276 39.960261, 116.378276 39.96032, 116.377202 39.960297, 116.377205 39.960191, 116.378477 39.960197, 116.378477 39.960438))

I'd try to build my polygon with JtsAdapter.java

JtsLayer layer = new JtsLayer(layerName, geoms); JtsMvt mvt = new JtsMvt(singletonList(layer)); return MvtEncoder.encode(mvt);

But when I debug my code ,I found that
final List<VectorTile.Tile.Feature> features JtsAdapter.toFeatures(layerGeoms, layerProps, userDataConverter); features's size is zero.
So,I follow the JtsAdapter.java and I found

 // Area must be non-zero
                final double exteriorArea = CGAlgorithms.signedArea(exteriorRing.getCoordinates());
                if(((int) Math.round(exteriorArea)) == 0) {
                    continue;
                }

the result of area is negative,and decimal,like -0.0008944.So Math.round(-0.0008944) ==0 is true.
Is a problem with that?

jts version is 1.13,
<dependency>
			<groupId>com.wdtinc</groupId>
			<artifactId>mapbox-vector-tile</artifactId>
			<version>2.0.0</version>
		</dependency>

Thanks for the great library!

According to the Vector tile specification (https://docs.mapbox.com/vector-tiles/specification/) original GeoJSON data is encoded into the proper tags of the vector tile protobuf.

Do we have methods to translate Vector Tile file into GeoJSON fiels ? It would great to be able to do that in the future

I want to add result layers to the android MapboxMap but I didn't find a way for do this. Can you help me?
I'm sorry if my English not very good!
Thanks in advance
`
MapView mapView = (MapView) findViewById(R.id.map_view);

    mapView.setStyleUrl(Style.SATELLITE_STREETS);

    mapView.onCreate(savedInstanceState);

    mapView.getMapAsync(new OnMapReadyCallback() {
        @RequiresApi(api = Build.VERSION_CODES.M)
        @Override
        public void onMapReady(MapboxMap mapboxMap) {
        //==================
        //not found any method for add JtsLayer to mapboxMap
        //==================
        }
    });`

I'm using LeafletJS with the Leaflet.VectorGrid plugin on the client and mapbox-vector-tile-java in the backend. The JTS geometry is fed in the backend by joining all polygons found in the envelope into a GeometryCollection and passing that to JtsAdapter.createTileGeom(). I hope that is the correct way to do it, the documentation is a little sparse on this.

Now, when I view my map in Leaflet I see these vertical and horizontal lines on the vector tile boundaries. I know 100% they are tile boundaries because they multiply when I zoom in.

My geometry is not intersected at the input level, so they geometry I put in the collection may exceed the envelope (but never be entirely outside it).

I have posted this to the Leaflet.VecctorGrid repository, they seem to think it's an issue in the backend (see github issue link). Any ideas on how to resolve this?

Leaflet/Leaflet.VectorGrid#108

Upgrade to latest JTS core release of 1.16.1.

Having a strange issue with the generation of mvt for osm building polygon. The mvt polygon contains additional segments. I am using the same code as in the testcase to build the mvt geom.

Original geometry

This is the mvt at zoom level 18

Zooming further you will find more segments getting added

The WKT is as follows

MULTIPOLYGON (((545718.1333394476 6866923.365958172, 545718.9237078322 6866923.839977133, 545720.1816180766 6866924.386922123, 545721.0721740036 6866924.623931634, 545721.9849938283 6866924.76978364, 545722.8978136531 6866924.806246642, 545724.2670433887 6866924.660394636, 545725.591745329 6866924.295764623, 545726.4377734606 6866923.912903129, 545727.6066281144 6866923.20187469, 545728.3190728544 6866922.600235288, 545728.9535939523 6866921.9439014485, 545729.5101914057 6866921.21464169, 545729.7550942858 6866920.813548851, 545730.1892402992 6866920.01136323, 545730.3673514862 6866919.5738074705, 545730.6456502129 6866918.698696026, 545730.834893346 6866917.805353191, 545730.9128169907 6866916.875547487, 545730.9128169907 6866916.419760417, 545730.834893346 6866915.508186351, 545730.6679141106 6866914.59661239, 545730.3896153839 6866913.721501486, 545730.0111291149 6866912.882853623, 545729.5435872523 6866912.080668792, 545728.9981217477 6866911.351409927, 545728.3636006496 6866910.676845537, 545727.2949335384 6866909.819966524, 545726.5045651537 6866909.345948387, 545725.2243910116 6866908.817235879, 545724.3338350846 6866908.598458301, 545722.9646053461 6866908.43437512, 545722.0517855214 6866908.470838053, 545721.1389656995 6866908.598458301, 545719.8142637592 6866908.981319068, 545718.9793675764 6866909.364179853, 545717.8995685162 6866907.522801962, 545735.9667218718 6866895.307731575, 545728.0964338719 6866883.566696301, 545707.9810018872 6866897.149106668, 545702.760117768 6866889.929460188, 545696.359247049 6866881.050762402, 545706.0774385931 6866873.411808483, 545703.1497359859 6866869.637912684, 545684.2365545015 6866845.244367993, 545640.4212029236 6866788.891551351, 545645.7868023807 6866777.515294569, 545645.4862397563 6866777.332982891, 545646.6773583078 6866775.38224821, 545648.1467755858 6866775.45517286, 545648.3026228724 6866775.291092397, 545621.6972645724 6866758.7554446785, 545621.6304728794 6866758.974218186, 545622.365181517 6866760.268628228, 545620.4727501745 6866763.34968959, 545617.2110890936 6866762.146434719, 545617.8010823934 6866758.627826799, 545619.2816316232 6866758.080893068, 545619.3484233161 6866757.880350706, 545588.3459451309 6866753.286108915, 545588.3682090285 6866753.450188931, 545589.4480080887 6866754.4346691165, 545588.9470703824 6866757.898581829, 545585.5740898103 6866758.080893068, 545584.6724019343 6866754.616980275, 545585.7744648921 6866753.559575613, 545585.7633329433 6866753.340802252, 545555.4732995008 6866761.30780264, 545555.5734870404 6866761.508345085, 545556.9538487276 6866761.927661125, 545557.8221407542 6866765.37334592, 545554.8387784028 6866766.868299566, 545552.6457844331 6866764.0060105035, 545553.2246457869 6866762.675137593, 545553.1244582445 6866762.474595118, 545528.411531287 6866781.72669545, 545528.6119063718 6866781.872544867, 545530.0034000052 6866781.635539561, 545532.2409217701 6866784.516066072, 545530.0367958545 6866787.104894575, 545526.9198501112 6866785.336469381, 545526.9532459578 6866783.896205848, 545526.7862667224 6866783.750356394, 545511.6802118206 6866811.188332018, 545511.8249271584 6866811.206563262, 545513.0605735079 6866810.495544769, 545516.255442893 6866812.191050503, 545515.242435526 6866815.472675521, 545511.6134201275 6866815.108050453, 545511.0902185207 6866813.777169095, 545510.8787114872 6866813.704244098, 545507.8730852351 6866844.916204217, 545508.0957242176 6866844.897972898, 545508.8638287046 6866843.676474521, 545512.4260524099 6866843.949944292, 545512.7934067302 6866847.450358159, 545510.4890932692 6866848.908864386, 545510.054947256 6866848.270767879, 545509.2311830248 6866848.817707736, 545511.4464408922 6866852.081116338, 545508.1068561665 6866854.360033744, 545501.1493879913 6866843.89525034, 545491.4311964471 6866850.403833543, 545480.6443377888 6866857.641675941, 545487.6574657082 6866868.142939957, 545480.9894282086 6866872.609627583, 545454.7180283823 6866833.339324052, 545446.8254764846 6866838.517013281, 545446.346802676 6866837.787761077, 545459.0483565745 6866829.200821329, 545458.5362869166 6866828.471569965, 545393.5034403963 6866872.354388225, 545420.8212434362 6866913.138094266, 545405.6038690461 6866923.402421167, 545415.3109286414 6866937.87824368, 545481.8243243913 6867037.09484589, 545478.2287048395 6867039.537901499, 545480.2547195709 6867042.527910863, 545484.0952420029 6867048.216224419, 545487.6908615547 6867045.7731661815, 545505.7914107596 6867072.7380067445, 545521.0533129451 6867062.473489517, 545531.16112271 6867055.600098971, 545644.5622879828 6866979.245693495, 545653.1784165691 6866973.356880924, 545715.7622342936 6866931.187274624, 545712.8122677859 6866926.811712037, 545715.3948799734 6866924.988561657, 545718.044283854 6866923.293032177, 545718.1333394476 6866923.365958172)))

It is a special case of LineString. I know in mapbox specification, the LinearRing should refer to Polygon. BUT in some case, a LineString(that's a LinearRing) should be draw as a closed ring. I think the implement following:

public static boolean shouldClosePath(VectorTile.Tile.GeomType geomType) {
        final boolean closeReq;

        switch(geomType) {
            case POLYGON:
                closeReq = true;
                break;
            default:
                closeReq = false;
                break;
        }

        return closeReq;
    }

should include the case of LinearRing.

Do you think it's a issue?

Thanks for this library!

I'm currently testing this to create MVT from a graphhopper routing engine storage. If I did nothing wrong your mapbox-vector-tile-java library is already roughly 10 times faster than https://github.com/ElectronicChartCentre/java-vector-tile which is really nice :) !

For bigger tiles fetching the geometry from the graphhopper storage takes around 0.5sec and this produces around 350 000 simple lines (a line going from junction to junction, i.e. with "from" and "to" Coordinate) and converting this into a MVT takes around 4 sec. See the experimental code I use (see update). Usually the lines gets more coordinates and are not just going straight from junction to junction, which I have neglect for now.

Is there a possibility to further improve speed via an index or using a Multiline or something else?

Update: See this work in progress

The README has pom.xml entries for 3.1.1, but that version is not on Maven Central: https://search.maven.org/search?q=a:mapbox-vector-tile

I run the jar in hadoop 2.7.1 and the version of protobuf in hadoop is 2.5.0.Can i or how to rebuild the project and generate new jars in protobuf 2.5.0

private static List<Geometry> flatIntersection(Geometry envelope, List<Geometry> dataGeoms) {
        final List<Geometry> intersectedGeoms = new ArrayList<>(dataGeoms.size());

        Geometry nextIntersected;
        for(Geometry nextGeom : dataGeoms) {
            try {

                // AABB intersection culling
                if(envelope.getEnvelopeInternal().intersects(nextGeom.getEnvelopeInternal())) {

                    nextIntersected = envelope.intersection(nextGeom);
                    if(!nextIntersected.isEmpty()) {
                        nextIntersected.setUserData(nextGeom.getUserData());
                        intersectedGeoms.add(nextIntersected);
                    }
                }

            } catch (TopologyException e) {
                LoggerFactory.getLogger(JtsAdapter.class).error(e.getMessage(), e);
            }
        }

        return intersectedGeoms;
    }

I cannot catch TopologyException because of this , and also cannot handle log about this.
plz throw exception to us Or just printStackTrace so that we can handle Exception, and it's log

I am getting the following error:
W/System.err: java.lang.ArrayIndexOutOfBoundsException: length=1; index=1 W/System.err: at org.locationtech.jts.geom.impl.CoordinateArraySequence.setOrdinate(CoordinateArraySequence.java:299) W/System.err: at com.wdtinc.mapbox_vector_tile.adapt.jts.MvtReader.readPoints(MvtReader.java:227) W/System.err: at com.wdtinc.mapbox_vector_tile.adapt.jts.MvtReader.readGeometry(MvtReader.java:164) W/System.err: at com.wdtinc.mapbox_vector_tile.adapt.jts.MvtReader.loadMvt(MvtReader.java:141) W/System.err: at com.wdtinc.mapbox_vector_tile.adapt.jts.MvtReader.loadMvt(MvtReader.java:97)
when I attempt to parse a tile with MULTILINESTRING
I suspect the geometry in this case is an array of arrays and this is causing the failure.

I am using HERE for tile data.

The tile where this occurs is:
https://vector.hereapi.com/v2/vectortiles/base/mc/16/10510/25420/omv?apiKey=xxyyzzzz

Is there a fix or workaround for this?

As discussed in #14, it would be great to support Android versions down to API level 15, if possible, as mentioned by @devemux86.

According to @snodnipper, it looks like we have some uses of the static methods in java.util.Objects, which are available above API 19 (Android 4.4.+).

I am not sure what else needs to be done and if these method calls can be easily converted to Android API level 15 compatibility? If it's just these Objects.requireNonNull calls, I can create a PR adding a Util Method like:

requireNonNull(Object obj)
   if(obj == null)
      throw new NullPointException();

Simplification is useful for a variety of reasons but there are situations where simplification is not desired. For example, at lower zoom levels or if wanting to edit features where maximum precision and accuracy is necessary. I'm happy to help with this but could use some help as the simplification seems to occur in the Affine Transform.