simonw / download-tiles Goto Github PK

View Code? Open in Web Editor NEW

28.0 4.0 6.0 29 KB

Download map tiles and store them in an MBTiles database

Home Page: https://datasette.io/tools/download-tiles

License: Apache License 2.0

Python 100.00%

mbtiles openstreetmap datasette-io datasette-tool

download-tiles's Introduction

download-tiles

Download map tiles and store them in an MBTiles database

Installation

Install this tool using pip:

pip install download-tiles

Usage

This tool downloads tiles from a specified TMS (Tile Map Server) server for a specified bounding box and range of zoom levels and stores those tiles in a MBTiles SQLite database. It is a command-line wrapper around the Landez Python libary.

Please use this tool responsibly. Consult the usage policies of the tile servers you are interacting with, for example the OpenStreetMap Tile Usage Policy.

Running the following will download zoom levels 0-3 of OpenStreetMap, 85 tiles total, and store them in a SQLite database called world.mbtiles:

download-tiles world.mbtiles

You can customize which tile and zoom levels are downloaded using command options:

--zoom-levels=0-3

The different zoom levels to download. Specify a single number, e.g. 15, or a range of numbers e.g. 0-4. Be careful with this setting as you can easily go over the limits requested by the underlying tile server.

--bbox=3.9,-6.3,14.5,10.2

The bounding box to fetch. Should be specified as min-lon,min-lat,max-lon,max-lat. You can use bboxfinder.com to find these for different areas.

--city=london

Or:

--country=madagascar

These options can be used instead of --bbox. The city or country specified will be looked up using the Nominatum API and used to derive a bounding box.

--show-bbox

Use this option to output the bounding box that was retrieved for the --city or --country without downloading any tiles.

--name=Name

A name for this tile collection, used for the name field in the metadata table. If not specified a UUID will be used, or if you used --city or --country the name will be set to the full name of that place.

--attribution="Attribution string"

Attribution string to bake into the metadata table. This will default to © OpenStreetMap contributors unless you use --tiles-url to specify an alternative tile server, in which case you should specify a custom attribution string.

You can use the --attribution=osm shortcut to specify the © OpenStreetMap contributors value without having to type it out in full.

--tiles-url=https://...

The tile server URL to use. This should include {z} and {x} and {y} specifiers, and can optionally include {s} for subdomains.

The default URL used here is for OpenStreetMap, http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png

--tiles-subdomains=a,b,c

A comma-separated list of subdomains to use for the {s} parameter.

--verbose

Use this option to turn on verbose logging.

--cache-dir=/tmp/tiles

Provide a directory to cache downloaded tiles between runs. This can be useful if you are worried you might not have used the correct options for the bounding box or zoom levels.

Databases created with this tool will have their SQLite application_id set to 0x4d504258, as described in the SQLite magic.txt file.

Development

To contribute to this tool, first checkout the code. Then create a new virtual environment:

cd download-tiles
python -mvenv venv
source venv/bin/activate

Or if you are using pipenv:

pipenv shell

Now install the dependencies and tests:

pip install -e '.[test]'

To run the tests:

pytest

download-tiles's People

Contributors

Stargazers

Watchers

Forkers

psiborg tmogi21a wuruilongll nicolaswurtz

download-tiles's Issues

Add error handling for HTTP requests

For folks with an unstable Internet connection, it is very annoying to have to restart the script every time an error is encountered. I suggest handling https requests with a try-except structure, with caught exceptions throwing a prompt such as Abort/Retry/Fail (suggested default: wait with exponential backoff, then retry up to 5 times).

Add new CLI option for custom request headers

When downloading from mapy.cz, any request will fail with 403: Forbidden.
My command (downloads hiking maps for most of the Czech Republic):
download-tiles cr.mbtiles --zoom-levels=6-15 --bbox="12.1398,48.6183,17.5231,50.9722" --attribution="" --tiles-url=https://mapserver.mapy.cz/turist-m/{z}-{x}-{y}

Turns out, Mapy.cz implemented protection against bulk downloads by only allowing requests with a specific referer. This can be circumvented by changing line 153 in cli.py from
tiles_headers={"User-Agent": user_agent},
to
tiles_headers={"Referer": "https://mapy.cz/"},

Could somebody implement a way to do this without modifying the source code? An option such as --header={"Name:" "Value"} would be appreciated. This is important because devs of their Android app keep killing support for older devices and their offline maps, requiring us to resort to measures like these. Anyway, I hope anyone seeking good offline hiking maps finds this useful.
Cheers, VT

--attribution mechanism

As of simonw/datasette-tiles#6 datasette-tiles will use the attribution value in the metadata table for the Leaflet map attribution.

This tool should provide an option to bake attribution into the generated file. It should default to using © OpenStreetMap contributors if the default OpenStreetMap tile provider is used.

--city and --country options as alternatives to --bbox

Looking up bounding boxes is a hassle. Nominatim provides an API, documented at https://nominatim.org/release-docs/latest/api/Search/ which can be used to look up the bounding boxes of specific addresses, cities and countries.

The city= and country= APIs are particularly relevant here:

--attribution=osm shortcut

If you use a different --tiles-url but it's ultimately sourced from OpenStreetMap having to type --attribution="© OpenStreetMap contributors" is a bit of a pain, especially since you need to type that ©. Instead, support --attribution=osm as a shortcut for that value.

(If anyone ever wants to explicitly set "osm" as the attribution they'll have to do so by running sqlite3 tiles.db "update metadata set value = 'osm' where name = 'attribution';" manually instead.)

--tiles-url is not respected

ModuleNotFoundError: No module named 'attr'

Got this error after installing with pipx install download-tiles

inat % download-tiles --help
Traceback (most recent call last):
  File "/Users/simon/.local/bin/download-tiles", line 5, in <module>
    from download_tiles.cli import cli
  File "/Users/simon/.local/pipx/venvs/download-tiles/lib/python3.9/site-packages/download_tiles/cli.py", line 1, in <module>
    from attr import attrib
ModuleNotFoundError: No module named 'attr'

Make sure this sets the application ID in the SQLite files it creates

https://www.sqlite.org/src/artifact?ci=trunk&filename=magic.txt

Suggests that PRAGMA application_id = INTEGER; be used to set this for MBTiles:

>68  belong  =0x4d504258  MBTiles tileset -

Add progress indicator

The program can easily calculate how many tiles need to be downloaded, and can obviously count successful downloads. Neither is displayed to the user, and a glance at the cache directory is not helpful as the tiles are downloaded in an apparently random order. Please add a number of finished & total downloads, percentage and/or progress bar on the command line to indicate progress and allow the user to guess the ETA. If you are concerned about output on the CLI for some reason, you could add an option to disable this.

Obviously, this will never be accurate since the file size for complex (city) tiles is orders of magnitude larger than simple (sea) tiles but could help greatly in estimating if the ETA is seconds or hours, especially given the exponential nature of downloading map tiles of various sizes.

--name option for populating the name field in metadata

It currently defaults to a UUID: https://github.com/makinacorpus/landez/blob/6e5c71ded6071158e7943df204cd7bd1ed623a30/landez/tiles.py#L338

If not specified, the --city and --country options could automatically populate it based on the JSON they get back from Nominatim.