kosmtik / kosmtik Goto Github PK
View Code? Open in Web Editor NEWMake maps with OpenStreetMap and Mapnik
License: Other
Make maps with OpenStreetMap and Mapnik
License: Other
To try the application, I've retrieved the osm-bright from @mapbox, loaded the local data using imposm but when I'm using the inspect tool, the rendering is OK whereas the "classic" rendering fails due to a behaviour related to tiles scheme.
See both images below to get an illustration
We certainly need to make them in sequential mode.
With openstreetmap-carto, I had a localconfig.json
[
{
"where": "Layer",
"if": {
"Datasource.type": "postgis"
},
"then": {
"Datasource.dbname": "bc_gis"
}
}
]
I then used git to switch from the master branch to the zoo branch (gravitystorm/openstreetmap-carto#1274). Kosmtik then started rendering from the default gis
database.
I've had trouble reproducing this, so could be some kind of race condition that happens with auto-reload.
When changing the metatile size, reload is needed to get the update...
I switched windows back to see this
[httpserver] /repo/poll/ 200
/usr/home/pnorman/osm/kosmtik/src/back/ProjectServer.js:18
if (filename.indexOf('.') === 0) return;
^
TypeError: Cannot call method 'indexOf' of null
at FSWatcher.<anonymous> (/usr/home/pnorman/osm/kosmtik/src/back/ProjectServer.js:18:26)
at FSWatcher.emit (events.js:98:17)
at FSEvent.FSWatcher._handle.onchange (fs.js:1051:12)
When I use the --localconfig
option with a relative path (e.g. ../openstreetmap-carto.localconfig.js
) I get the output [Local Config] Unable to load local config from ../openstreetmap-carto.localconfig.js
. If I use the absolute path (e.g. ~/path/to/openstreetmap-carto.localconfig.js
) this works as expected.
I'd like to be able to install multiple plugins in one command line, rather than installing them one-by-one.
In the top right there's a pair of arrows pointing up and down. It doesn't appear to be a button, and I'm not sure what it is!
Perhaps some mouse-over text would help
When you click on the inspect view, you only get on feature returned, even if there are multiple features overlapping or within a pixel or two from the cursor.
It would be great to show the multiple features in the popup, and well as #28
I'm converting a vector tile based style (vector tile to raster) to Mapnik XML. At time of conversion, the TileJSON for the vector tiles does not exist and Kosmtik gives the error
[Core] Using mapnik version 3.0.0
/data/servers/mq_nms/kosmtik/src/plugins/datasource-loader/index.js:20
if (err) throw err;
^
Error: connect ECONNREFUSED
at exports._errnoException (util.js:746:11)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:983:19)
I was wondering
Some weeks ago (I don't remember exactly when), I was able to load plugins when doing a fresh install.
Now, when I install all plugins e.g
node index.js plugins --install kosmtik-deploy
node index.js plugins --install kosmtik-fetch-remote
node index.js plugins --install kosmtik-geojson-overlay
node index.js plugins --install kosmtik-map-compare
node index.js plugins --install kosmtik-mbtiles-export
node index.js plugins --install kosmtik-open-in-josm
node index.js plugins --install kosmtik-osm-data-overlay
node index.js plugins --install kosmtik-overlay
node index.js plugins --install kosmtik-overpass-layer
node index.js plugins --install kosmtik-place-search
node index.js plugins --install kosmtik-tiles-export
Then I do node index.js plugins --available
and get
[Core] Loading config from /root/kosmtik/projects/OSMBright/localconfig.json
[Core] Loading plugin from ../plugins/base-exporters/index.js
[Core] Loading plugin from ../plugins/hash/index.js
[Core] Loading plugin from ../plugins/local-config/index.js
[Core] Loading plugin from ../plugins/datasource-loader/index.js
Loading available plugins…
. kosmtik-deploy (Deploy your local Kosmtik project on a remote server)
. kosmtik-fetch-remote (Kosmtik plugin to fetch remote layer data)
. kosmtik-geojson-overlay (Show a geojson overlay on top of your Kosmtik map)
. kosmtik-map-compare (Add a map to compare side-by-side with your work)
. kosmtik-mbtiles-export (Export your Kosmtik project in MBTiles)
. kosmtik-open-in-josm (Add a control to open JOSM with current map view)
. kosmtik-osm-data-overlay (Display an interactive OSM data overlay in Kosmtik)
. kosmtik-overlay (Add an overlay to Kosmtik)
. kosmtik-overpass-layer (Add overpass layers to your Kosmetik project)
. kosmtik-place-search (Search for places in Kosmtik)
. kosmtik-tiles-export (Export tiles tree from your Kosmtik project)
and when I do node index.js plugins --installed
, I only get the following
[Core] Loading config from /root/kosmtik/projects/OSMBright/localconfig.json
[Core] Loading plugin from ../plugins/base-exporters/index.js
[Core] Loading plugin from ../plugins/hash/index.js
[Core] Loading plugin from ../plugins/local-config/index.js
[Core] Loading plugin from ../plugins/datasource-loader/index.js
Installed plugins
../plugins/base-exporters/index.js
../plugins/hash/index.js
../plugins/local-config/index.js
../plugins/datasource-loader/index.js
when doing a git pull
[Core] Loading project from /home/pnorman/osm/openstreetmap-carto/repo/project.yaml
[Core] Generating Mapnik XML…
[Core] Using mapnik version 2.2.0
[Core] Loading map…
[Core] Map ready
[Core] Loading map…
[Core] Map ready
[httpserver] /repo/reload/ 200
[httpserver] /repo/poll/ 200
file could not be found: '/home/pnorman/osm/openstreetmap-carto/repo/symbols/picnic.p.16.png' in style 'amenity-points-poly' in PointSymbolizer at line 14543
[httpserver] /repo/tile/17/70544/43861.xray?t=1425253156811&layer=landuse-overlay&background= 500
/usr/home/pnorman/osm/kosmtik/src/back/Tile.js:39
map.zoomToBox(this.projection.forward([this.minX, this.minY, this.maxX, th
^
TypeError: Cannot call method 'zoomToBox' of null
at Tile.renderToVector (/usr/home/pnorman/osm/kosmtik/src/back/Tile.js:39:9)
at /usr/home/pnorman/osm/kosmtik/src/back/ProjectServer.js:124:21
at dispense (/usr/home/pnorman/osm/kosmtik/node_modules/generic-pool/lib/generic-pool.js:250:16)
at Object.me.release (/usr/home/pnorman/osm/kosmtik/node_modules/generic-pool/lib/generic-pool.js:349:5)
at release (/usr/home/pnorman/osm/kosmtik/src/back/ProjectServer.js:120:55)
at ProjectServer.raise (/usr/home/pnorman/osm/kosmtik/src/back/ProjectServer.js:218:13)
at /usr/home/pnorman/osm/kosmtik/src/back/ProjectServer.js:121:30
at /usr/home/pnorman/osm/kosmtik/node_modules/generic-pool/lib/generic-pool.js:274:11
at loaded (/usr/home/pnorman/osm/kosmtik/src/back/MapPool.js:24:37)
Segmentation fault (core dumped)
I am seeing a test failure:
2) #Tile() #renderToVector() should render a PBF:
Uncaught AssertionError: [26,231,1,10,5,119,111,114,108,100,18,106,8,45,18,2,0,0,24,3,34,96,9,138,16,144,27,202,1,182,1,133,2,240,8,6,222,6,142,3,128,3,3 deepEqual [26,131,2,10,5,119,111,114,108,100,18,106,8,45,18,2,0,0,24,3,34,96,9,138,16,144,27,202,1,182,1,133,2,240,8,6,222,6,142,3,128,3,3
at /home/tom/rpm/BUILD/package/test/tile.js:62:24
Dumping both arrays fully shows that there are a number of minor numeric differences that may be precisions issues, but also a whole chunk of numbers missing compared to the expected result. I am getting 234 entries instead of the expected 262 entries.
It seems that node-mapnik
only tests the length of the arrays returned by this method, presumably because figuring out the precision issues is hard... Doesn't explain why the arrays are different lengths though - maybe a mapnik version issue as I am on 2.2.0 still?
[httpserver] /src/front/fonts/DejaVuSans-webfont.woff 200
/home/pnorman/kosmtik/src/back/ProjectServer.js:102
var content = tile.getData();
^
TypeError: Object #<Image> has no method 'getData'
at /home/pnorman/kosmtik/src/back/ProjectServer.js:102:32
Segmentation fault (core dumped)
For what I understand now, we have those subtasks:
Though, many areas of the process remain confused for me at this point.
cc @pnorman in case you want to 1. add some elements 2. track the progress of the issue
I'm interested by the topic and would like to contribute. I didn't see a reference to this feature in the code to start from something.
Do you have already ideas for managing it? I can make some search if not.
When used behind a proxy, kosmtik is unable to download the necessary tiles to display the map. a proxy option should be included.
Probably the same issue as #10
/home/pnorman/kosmtik/node_modules/generic-pool/lib/generic-pool.js:316
throw new Error("pool is draining and cannot accept work");
^
Error: pool is draining and cannot accept work
at Object.me.acquire (/home/pnorman/kosmtik/node_modules/generic-pool/lib/generic-pool.js:316:13)
at ProjectServer.tile (/home/pnorman/kosmtik/src/back/ProjectServer.js:50:18)
at ProjectServer.serveTile (/home/pnorman/kosmtik/src/back/ProjectServer.js:45:15)
at tile (/home/pnorman/kosmtik/src/back/ProjectServer.js:35:108)
at StateBase.when (/home/pnorman/kosmtik/src/back/StateBase.js:51:33)
at ProjectServer.serve (/home/pnorman/kosmtik/src/back/ProjectServer.js:35:70)
at PreviewServer.forwardToProject (/home/pnorman/kosmtik/src/back/PreviewServer.js:73:23)
at PreviewServer.serve (/home/pnorman/kosmtik/src/back/PreviewServer.js:67:42)
at Server.EventEmitter.emit (events.js:98:17)
at HTTPParser.parser.onIncoming (http.js:2108:12)
pnorman@pnorman-desktop:~/kosmtik$
I don't know exactly when this happened, but probably when saving a file and triggering a reload.
When I do
node index.js plugins --install kosmtik-map-compare
I get below logs
[Core] Loading config from /root/kosmtik/projects/OSMBright/localconfig.json
[Core] Loading plugin from ../plugins/base-exporters/index.js
[Core] Loading plugin from ../plugins/hash/index.js
[Core] Loading plugin from ../plugins/local-config/index.js
[Core] Loading plugin from ../plugins/datasource-loader/index.js
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] No README data
[email protected] node_modules/kosmtik-map-compare
[Core] Successfully installed package kosmtik-map-compare
[Core] Attached plugin: kosmtik-map-compare
[Core] Saved env conf to /root/kosmtik/projects/OSMBright/localconfig.json
The issue is the fact that doing cat projects/OSMBright/localconfig.json
returns a yml notation. Then, when restarting, it fails because the json was replaced with the yml content e.g
- where: center
then:
- 29.9377
- -3.4216
- 2
FIY, I've set export KOSMTIK_CONFIGPATH=/root/kosmtik/projects/OSMBright/localconfig.json
Without a $HOME/.config
directory, kosmtik was unable to install a plugin. When I added it, it was.
The online carto docs these days are a bit rudimentary and really hard to browse.
It'd be good to have a Carto reference available within kosmtik (as it is within Tilemill), perhaps as a plugin.
I was trying to export an image of what was displaying in my map (which is below):
And this was my exported image:
It looks like the correct zoom level wasn't used. ;)
Here's the my mml file: https://gist.github.com/218370ad7ce67f95edd4
At the moment, the only feedback about the process is the number of tiles to be created by zoom.
We need something like an ETA.
At present the workflow for uploading a completed map to a tileserver is a bit painful:
Ideal workflow:
Currently, running node index.js serve test/data/minimalist-project.mml
or test/data/world/project.yml
only leads to
[Local Config] Unable to load local config from /.../kosmtik/kosmtik/test/data/world/localconfig.js
[Error: File not found: /.../kosmtik/kosmtik/test/data/world/localconfig.js]
but seeing kosmtik in action right after pulling would be nice.
Unfortunately land-low.shp
and consorts aren't available, too.
So we would have "Data inspector: show land layer", "Data inspector: show waterway layer", etc.
in Firefox (37, Ubuntu 12) exporting images as png and xml etc all download as a file with no filename. In Chromium files are "download.png" with the correct extension.
The following hack seems to fix it. Note that it uses setHeader instead of writeHead. If your project is "osm-carto" in firefox and chrome it will give the download a filename such as osm-carto.png.
Can add this to a PR if it looks okay?
ProjectServer.prototype.export = function (res, options) {
var filename = this.project.id + "." + options.format.substr(0,3);
this.project.export(options, function (err, buffer) {
if (err) return self.raise(err.message, res);
res.setHeader('Content-disposition', 'attachment; filename='+filename);
res.write(buffer);
res.end();
});
};
Hello,
I am seeing a seg fault when turning on the Data Inspector when the tile is not showing anything. This is using current master ee1fd3e
httpserver] /osm-bright/tile/10/504/334.png?t=1426111602116 200
[httpserver] /osm-bright/tile/10/504/338.png?t=1426111602116 200
[httpserver] /osm-bright/tile/10/508/336.xray?t=1426111672909&layer=&background= 200
[httpserver] /osm-bright/tile/10/508/335.xray?t=1426111672910&layer=&background= 200
[httpserver] /osm-bright/tile/10/507/334.xray?t=1426111672910&layer=&background= 200
[httpserver] /osm-bright/tile/10/507/335.xray?t=1426111672910&layer=&background= 200
[httpserver] /osm-bright/tile/10/507/336.xray?t=1426111672910&layer=&background= 200
/home/tim/projects/kosmtik/src/back/XRayTile.js:18
vtile.setData(this.data);
^
Error: cannot accept empty buffer as protobuf
at XRayTile.render (/home/tim/projects/kosmtik/src/back/XRayTile.js:18:11)
at /home/tim/projects/kosmti
It is able to show the data tiles when there is something rendered on each tile, but appears to happen when there is an empty tile. The background of the tiles are transparent, if that helps, with the project.mml living at https://github.com/timwaters/cloud_mapping/blob/master/osm-bright/osm-bright/project.mml
When you turn on the inspection view, and click on a feature, you get a popup. It would be useful if that popup indicated which layer the feature was found in.
There's no option to bind kosmtik to a specific IP. This is a problem if you either just want to run it locally and not have it outside accessible, or are on a machine with multiple IPs and want to bind it to one specifically.
To use current view as bbox for export.
The title of the kosmtik interface page is "Testing". This should be something else.
I am seeing a test failure:
1) #Tile() #renderToVector() should render a GeoJSON:
Uncaught AssertionError: {"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[-71.71188354492186,19.71 == [123,34,116,121,112,101,34,58,34,70,101,97,116,117,114,101,67,111,108,108,101,99,116,105,111,110,34,44,34,102,101,97,116,117,114
at /home/tom/rpm/BUILD/package/test/tile.js:53:24
There appear to be two problems - firstly the hash keys are not necessarily ordered the same when the first argument to the assert is formatted as in the example output. Using deepEqual
should fix this I think.
The second is that some of the coordinates in the GeoJSON differ in the last decimal place, due to floating point comparison issues. There are ways to deal with this for simple comparisons of numeric values but I haven't yet found one that applies to numeric comparisons done as part of a deep comparison of a data structure...
When working on a layer, it can be helpful to only render this one (even if in some cases it can render a bit differently - for example labels). This should be simpler than "unchecking" all other layers.
I had an error in my localconfig.json and got the error
undefined:11
]
^
SyntaxError: Unexpected token ]
at Object.parse (native)
at L.fromString (/usr/home/pnorman/osm/kosmtik/node_modules/json-localizer/index.js:71:46)
at /usr/home/pnorman/osm/kosmtik/src/plugins/local-config/index.js:38:15
at fs.js:272:14
at Object.oncomplete (fs.js:108:15)
Instead of being told the filename, it is "undefined".
When exporting tiles with the module kosmtik-tiles-export and the option minZoom = 0, kosmtik fails with segmentation fault.
Hi all,
this morning I added few (wrong) lines to style.mss to display a border as dotted:
#boundary[name='Italy'] {
line-dasharray: [1, 1];
}
and I got this error:
node index.js serve --host 0.0.0.0 --proxy="http://www.proxy.org:8080" --mapnik-version=2.2.0 /project.mml
[Core] Loading config from /Users/denix/.config/kosmtik.yml
[Core] Loading plugin from ../plugins/base-exporters/index.js
[Core] Loading plugin from ../plugins/hash/index.js
[Core] Loading plugin from ../plugins/local-config/index.js
[Core] Loading plugin from ../plugins/datasource-loader/index.js
[Core] Loading plugin from kosmtik-fetch-remote
[Core] Loading plugin from kosmtik-overpass-layer
[Core] Loading plugin from kosmtik-place-search
[Core] Loading plugin from kosmtik-overlay
[Core] Loading plugin from kosmtik-osm-data-overlay
[Core] Loading plugin from kosmtik-open-in-josm
[Core] Loading plugin from kosmtik-map-compare
[Core] Loading plugin from kosmtik-deploy
[Core] Loading plugin from kosmtik-tiles-export
[Core] Loading project from /project.mml
[Core] PreviewServer started, you can browse http://0.0.0.0:6789
[Fetch Remote] Done.
[Overpass Layer] File already exists and not force mode /data/countries.geojson Skipping
[Overpass Layer] File already exists and not force mode /data/cities.geojson Skipping
[Core] Generating Mapnik XML…
[Core] Using mapnik version 2.2.0
/Volumes/external/git/kosmtik/node_modules/carto/lib/carto/renderer.js:152
if (env.errors) throw env.errors;
^
Error: style.mss:27:4 Invalid value for line-dasharray, the type numbers is expected. [1,1] (of type field) was given.
at Object.env.error (/Volumes/external/git/kosmtik/node_modules/carto/lib/carto/parser.js:235:55)
at Rule.tree.Rule.toXML (/Volumes/external/git/kosmtik/node_modules/carto/lib/carto/tree/rule.js:83:24)
at Definition.tree.Definition.symbolizersToXML (/Volumes/external/git/kosmtik/node_modules/carto/lib/carto/tree/definition.js:139:46)
at Definition.tree.Definition.toXML (/Volumes/external/git/kosmtik/node_modules/carto/lib/carto/tree/definition.js:200:29)
at /Volumes/external/git/kosmtik/node_modules/carto/lib/carto/tree/style.js:31:27
at Array.map (native)
at Object.tree.StyleXML (/Volumes/external/git/kosmtik/node_modules/carto/lib/carto/tree/style.js:30:29)
at Renderer.render (/Volumes/external/git/kosmtik/node_modules/carto/lib/carto/renderer.js:134:39)
at Carto.render (/Volumes/external/git/kosmtik/src/back/renderer/Carto.js:20:45)
at Project.render (/Volumes/external/git/kosmtik/src/back/Project.js:57:25)
that make Kosmtik to crash. The right behavior should be to display the error and continue (I think).
Best.
I'm working on a Full HD display and the map in Kosmtik looks blurry while the map on osm.org looks perfectly crisp.
Ref: https://lists.openstreetmap.org/pipermail/dev/2015-April/028504.html
In the inspection view, all the data is the same (orange) colour. It would be helpful to colour each layer seperately, so that it's easier to tell which layer different features are coming from.
mapbox-studio has a great "xray" behaviour that generates colours automatically for each layer. It also fills in polygons, draws and labels point features.
Different carto versions between production and development recently caused problems (gravitystorm/openstreetmap-carto#1294).
How would I install/run Kosmtik to make use of a specific version of carto?
I am trying to bundle up kosmtik into a tarball, and would like to pre-install some plugins. The issue I am having is that the plugins need to be listed in $HOME/.config/kosmtik.yml
to be usable, and there's no reasonable way of bundling that.
Is there a way to do what I want?
The data inspector images aren't refreshed when the source files change. The map images refresh, and turning the data inspector off and on again is a workaround.
Following the instructions in the readme, I got
$ node index.js -h
module.js:340
throw err;
^
Error: Cannot find module 'npm'
at Function.Module._resolveFilename (module.js:338:15)
at Function.Module._load (module.js:280:25)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object.<anonymous> (/home/pnorman/osm/kosmtik/src/back/PluginsManager.js:1:73)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
Versions were
pnorman@merry:~/osm/kosmtik$ npm --version
1.4.28
pnorman@merry:~/osm/kosmtik$ nodejs --version
v0.10.32
A npm install npm
fixed this, but there's probably a more appropriate way.
[httpserver] /mq-carto-style.tm2/reload/ 200
/home/pnorman/kosmtik/src/back/Utils.js:20
return str.replace(/\{ *([\w_]+) *\}/g, function (str, key) {
^
TypeError: Cannot call method 'replace' of undefined
at Object.module.exports.template (/home/pnorman/kosmtik/src/back/Utils.js:20:20)
at VectorBasedTile._render (/home/pnorman/kosmtik/src/back/VectorBasedTile.js:35:24)
at VectorBasedTile.render (/home/pnorman/kosmtik/src/back/VectorBasedTile.js:44:10)
at /home/pnorman/kosmtik/src/back/ProjectServer.js:55:21
at /home/pnorman/kosmtik/node_modules/generic-pool/lib/generic-pool.js:281:11
at loaded (/home/pnorman/kosmtik/src/back/MapPool.js:28:28)
Occurred when saving a file from my editor.
Postgis Plugin: ERROR: column "the_geom" does not exist
LINE 1: SELECT ST_AsBinary("the_geom") AS geom,"name","rclass" FROM ...
^
in executeQuery Full sql was: 'SELECT ST_AsBinary("the_geom") AS geom,"name","rclass" FROM (SELECT name, rclass FROM vector_linear_labels
WHERE (rclass IN (0, 1, 2, 3, 12, 16) AND z(272989) >= 10) -- major roads
OR (rclass IN (4, 5, 6, 14, 11) AND z(272989) >= 14) -- minor roads, rivers, ferries, railways
ORDER BY path_priority DESC, rclass DESC
) AS trans_linear_labels WHERE "the_geom" && ST_SetSRID('BOX3D(-8419080.043442452 5024052.995128065,-8389728.224580945 5053404.813989572)'::box3d, 900913)'
[httpserver] /mq-carto-style.tm2source/tile/11/594/766.pbf 500
TypeError: first argument must be a buffer object
at Request.onResponse [as _callback] (/home/pnorman/kosmtik/src/back/VectorBasedTile.js:18:19)
at Request.self.callback (/home/pnorman/kosmtik/node_modules/request/request.js:372:22)
at Request.EventEmitter.emit (events.js:98:17)
at Request.<anonymous> (/home/pnorman/kosmtik/node_modules/request/request.js:1310:14)
at Request.EventEmitter.emit (events.js:117:20)
at IncomingMessage.<anonymous> (/home/pnorman/kosmtik/node_modules/request/request.js:1258:12)
at IncomingMessage.EventEmitter.emit (events.js:117:20)
at _stream_readable.js:920:16
at process._tickCallback (node.js:415:13)
Segmentation fault (core dumped)
The invalid SQL is my fault, but it shouldn't core dump
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.