Code Monkey home page Code Monkey logo

overpass-api's Introduction

How to use this image

By default, this image will clone an existing Overpass server for the whole planet, and make it available at http://localhost/api/interpreter.

The following enviroment variables can be used to customize the setup:

  • OVERPASS_MODE - takes the value of either init or clone. Defaults to clone.
  • OVERPASS_META - (init mode only) yes, no or attic - passed to Overpass as --meta or --keep-attic.
  • OVERPASS_PLANET_URL - (init mode only) url to a "planet" file (e.g. https://planet.openstreetmap.org/planet/planet-latest.osm.bz2)
  • OVERPASS_CLONE_SOURCE - (clone mode only) the url to clone a copy of Overpass from. Defaults to https://dev.overpass-api.de/api_drolbr/, which uses minute diffs.
  • OVERPASS_DIFF_URL - url to a diff directory for updating the instance (e.g. https://planet.openstreetmap.org/replication/minute/).
  • OVERPASS_COMPRESSION - (init mode only) takes values of no, gz or lz4. Specifies compression mode of the Overpass database. Defaults to gz.
  • OVERPASS_RULES_LOAD - integer, desired load from area generation. Controls the ratio of sleep to work. A value of 1 will make the system sleep 99x times longer than it works, a value of 50 will result in sleep and work in equal measure, and a value of 100 will only sleep 3 seconds between each execution. Defaults to 1.
  • OVERPASS_UPDATE_SLEEP - integer, the delay between updates (seconds).
  • OVERPASS_COOKIE_JAR_CONTENTS - cookie-jar compatible content to be used when fetching planet.osm files and updates.
  • OVERPASS_PLANET_PREPROCESS - commands to be run before passing the planet.osm file to update_database, e.g. conversion from pbf to osm.bz2 using osmium.
  • USE_OAUTH_COOKIE_CLIENT - set to yes if you want to use oauth_cookie_client to update cookies before each update. Settings are read from /secrets/oauth-settings.json. Read the documentation here.
  • OVERPASS_FASTCGI_PROCESSES - number of fcgiwarp processes. Defaults to 4. Use higher values if you notice performance problems.
  • OVERPASS_RATE_LIMIT - set the maximum allowed number of concurrent accesses from a single IP address.
  • OVERPASS_TIME - set the maximum amount of time units (available time).
  • OVERPASS_SPACE - set the maximum amount of RAM (available space) in bytes.
  • OVERPASS_MAX_TIMEOUT - set the maximum timeout for queries (default: 1000s). Translates to send/recv timeout for fastcgi_wrap.
  • OVERPASS_USE_AREAS - if false initial area generation and the area updater process will be disabled. Default true.
  • OVERPASS_HEALTHCHECK - shell commands to execute to verify that image is healthy. exit 1 in case of failures, exit 0 when container is healthy. Default healthcheck queries overpass and verifies that there is reponse returned
  • OVERPASS_STOP_AFTER_INIT - if false the container will keep runing after init is complete. Otherwise container will be stopped after initialization process is complete. Default true
  • OVERPASS_ALLOW_DUPLICATE_QUERIES - if yes, duplicate queries (same query from the same IP address) will be allowed. Default no.

Modes

Image works in two modes init or clone. This affects how the instance gets initialized:

  • init - OSM data is downloaded from OVERPASS_PLANET_URL, which can be a full planet or partial planet dump. This file will then be indexed by Overpass and later updated using OVERPASS_DIFF_URL.
  • clone - data is copied from an existing server, given by OVERPASS_CLONE_SOURCE, and then updated using OVERPASS_DIFF_URL. This mode is faster to set up, as the OSM planet file is already indexed. The default clone source provides an Overpass instance using minute diffs covering the whole world (hourly or daily diffs will not work with this image).

Running

To monitor the progress of file downloads, run with the stdin (-i) and TTY (-t) flags: docker run -i -t wiktorn/overpass-api

After initialization is finished, the Docker container will stop. Once you start it again (with docker start command) it will start downloading diffs, applying them to database, and serving API requests.

The container exposes port 80. Map it to your host port using -p: docker run -p 80:80 wiktorn/overpass-api

The Overpass API will then be available at http://localhost:80/api/interpreter.

Container includes binaries of pyosmium (in /app/venv/bin/) and osmium-tool (in /usr/bin)

All data resides within the /db directory in the container.

For convenience, a docker-compose.yml template is included.

Examples

Overpass instance covering part of the world

In this example the Overpass instance will be initialized with a planet file for Monaco downloaded from Geofabrik. Data will be stored in folder/big/docker/overpass_db/ on the host machine and will not contain metadata as this example uses public Geofabrik extracts that do not contain metadata (such as changeset and user). Overpass will be available on port 12345 on the host machine.

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/monaco-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/monaco/minute/ \
  -e OVERPASS_RULES_LOAD=10 \
  -v /big/docker/overpass_db/:/db \
  -p 12345:80 \
  -i -t \
  --name overpass_monaco wiktorn/overpass-api

Overpass clone covering whole world

In this example Overpass instance will be initialized with data from main Overpass instance and updated with master planet diffs. Data will be stored in /big/docker/overpass_clone_db/ on the host machine and the API will be available on port 12346 on the host machine.

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -v /big/docker/overpass_clone_db/:/db \
  -p 12346:80 \
  -i -t \
  --name overpass_world \
  wiktorn/overpass-api

Overpass instance covering part of the world using cookie

In this example Overpass instance will be initialized with planet file for Monaco downloaded from internal Geofabrik server. Data will be stored in /big/docker/overpass_db/ on the host machine and the API will be available on port 12347 on the host machine.

Prepare file with your credentials /home/osm/oauth-settings.json:

{
  "user": "your-username",
  "password": "your-secure-password",
  "osm_host": "https://www.openstreetmap.org",
  "consumer_url": "https://osm-internal.download.geofabrik.de/get_cookie"
}

Because Geofabrik provides only PBF extracts with metadata, osmium is used in OVERPASS_PLANET_PREPROCESS to convert the pbf file to osm.bz2 that's used by Overpass.

docker run \
    -e OVERPASS_META=yes \
    -e OVERPASS_MODE=init \
    -e OVERPASS_PLANET_URL=https://osm-internal.download.geofabrik.de/europe/monaco-latest-internal.osm.pbf \
    -e OVERPASS_DIFF_URL=https://osm-internal.download.geofabrik.de/europe/monaco-updates/ \
    -e OVERPASS_RULES_LOAD=10 \
    -e OVERPASS_COMPRESSION=gz \
    -e OVERPASS_UPDATE_SLEEP=3600 \
    -e OVERPASS_PLANET_PREPROCESS='mv /db/planet.osm.bz2 /db/planet.osm.pbf && osmium cat -o /db/planet.osm.bz2 /db/planet.osm.pbf && rm /db/planet.osm.pbf' \
    -e USE_OAUTH_COOKIE_CLIENT=yes \
    --mount type=bind,source=/home/osm/oauth-settings.json,target=/secrets/oauth-settings.json \
    -v /big/docker/overpass_db/:/db \
    -p 12347:80 \
    -i -t \
    --name overpass_monaco wiktorn/overpass-api

Healthcheck checking that instance is up-to-date

Using following environment variable:

-e OVERPASS_HEALTHCHECK='
  OVERPASS_RESPONSE=$(curl --noproxy "*" -s "http://localhost/api/interpreter?data=\[out:json\];node(1);out;" | jq -r .osm3s.timestamp_osm_base)
  OVERPASS_DATE=$(date -d "$OVERPASS_RESPONSE" +%s)
  TWO_DAYS_AGO=$(($(date +%s) - 2*86400)) ;
  if [ ${OVERPASS_DATE} -lt ${TWO_DAYS_AGO} ] ; then
    echo "Overpass out of date. Overpass date: ${OVERPASS_RESPONSE}"
    exit 1;
  fi
  echo "Overpass date: ${OVERPASS_RESPONSE}"
'

healthcheck will verify the date of last update of Overpass instance and if data in instance are earlier than two days ago, healthcheck will fail.

How to use Overpass after deploying using above examples

The Overpass API will be exposed on the port exposed by docker run - for example http://localhost:12346/api/interpreter.

You may then use this directly as an Overpass API url, or use it within Overpass Turbo.

Try a direct query with http://localhost:12346/api/interpreter?data=node(3470507586);out geom;, which should return a pub in Dublin.

To use the API in Overpass Turbo, go to settings and set Server tohttp://localhost:12346/api/. Now you will use your local Overpass instance for your queries.

overpass-api's People

Contributors

arichnad avatar cezarytarnowski-tomtom avatar cjxd avatar galewis2 avatar jalessio avatar jameschevalier avatar kai-inthesky avatar michieldemey avatar moeflon avatar nilsnolde avatar wiktorn avatar wiktorsikora avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

overpass-api's Issues

Error when attempting to use attic data

Hi,
For some reason, I can't get get queries to work that use the date (attic) feature.

For wget -O - "http://192.168.1.147:12345/api/interpreter?data=[date:\"2017-10-28T19:20:00Z\"];node[\"addr:housenumber\"];out;"

I get the following result/error:

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.55 579b1eec">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2018-04-28T07:13:02Z"/>
<remark> runtime error: open64: 2 No such file or directory /db/db/node_tags_global_attic.bin File_Blocks::File_Blocks::1 </remark>
</osm>

By removing [date:\"2017-10-28T19:20:00Z\"]; from the query, OSM data is returned.

Could someone have a look at this issue?
Thank you

Steps to reproduce:

I followed the README example to start an overpass-api instance for Monaco with OVERPASS_META=attic.
I also tried OVERPASS_META=yes, but it made no difference.

docker run \
  -e OVERPASS_META=attic \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/monaco-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/monaco/minute/ \
  -e OVERPASS_PLANET_SEQUENCE_ID=2946200 \
  -e OVERPASS_RULES_LOAD=10 \
  -v /data/docker/overpass_db/:/db \
  -p 12345:80 \
  -i -t \
  --name overpass_monaco wiktorn/overpass-api

docker start overpass_monaco

wget -O - "http://192.168.1.147:12345/api/interpreter?data=[date:\"2017-10-28T19:20:00Z\"];node[\"addr:housenumber\"];out;"

wget -O - "http://192.168.1.147:12345/api/interpreter?data=node[\"addr:housenumber\"];out;"


Rationale behind having to start the container twice?

I see that after the database is initialized, the script exits and tells you to run 'docker start', such that on the second run it will start nginx.

Is there a reason for that, rather than just starting nginx after initialisation?

No such file or directory /db/db//osm3s_v0.7.55_areas Unix_Socket::7

Hello! First of all thank you for this work and for your support ๐Ÿ‘

I have this issue with the container and that prevents areas queries to work. I read here and there and you said multiple times that areas are calculated as part of the execution o the container. In the initialization phase, I have seen multiple times messages about different parts of the "pipeline" being processed but, once it stopped and I restarted it, I see all "good" messages on the stdout/stderr except one nasty line:

runtime error: open64: 2 No such file or directory /db/db//osm3s_v0.7.55_areas Unix_Socket::7

Of course such file does not exist. Is there a way to trigger a make-area stage? Is there a way to see whether there's one running? How can I best troubleshoot why this file is not there and why I get this error? Thanks!

I tried the following command:

curl -g 'http://localhost:12345/api/interpreter?data=[out:json];area[name="Berlin"];out;'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
  <meta http-equiv="content-type" content="text/html; charset=utf-8" lang="en"/>
  <title>OSM3S Response</title>
</head>
<body>

<p>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</p>
<p><strong style="color:#FF0000">Error</strong>: runtime error: open64: 2 No such file or directory /db/db//osm3s_v0.7.55_areas Unix_Socket::7 </p>

</body>
</html>

I have bootstrapped the container with only Germany and the rules_loop.log is filled with "updates started/finished" messages.

Log Nginx Errors

Sometimes I receive wird Nginx errors when accessing the Overpass instance. The error contains a hint to check the Nginx error log, but I don't think the container is writing logs? Maybe it would be possible to also write the Nginx logs to the disk?

Failed to process planet file

[ ok ]
osm:~# service docker start

  • Starting docker ...
    [ ok ]
    osm:~# docker run \
-e OVERPASS_META=yes \
-e OVERPASS_MODE=init \
-e OVERPASS_PLANET_URL=https://download.geofabrik.de/europe/germany/baden-

wuerttemberg-latest.osm.bz2 \

-e OVERPASS_DIFF_URL=http://download.geofabrik.de/europe/germany/baden-wue

rttemberh-updates/ \

-e OVERPASS_RULES_LOAD=10 \
-v /big/docker/overpass_db/:/db \
-i -t \
--name overpass_bw wiktorn/overpass-api

No database directory. Initializing
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 717M 100 717M 0 0 1708k 0 0:07:09 0:07:09 --:--:-- 1602k
[======================================================================] 100%
Reading XML file ...terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
/app/bin/init_osm3s.sh: line 44: 15 Broken pipe bunzip2 < $PLANET_FE
16 Aborted | $EXEC_DIR/bin/update_database --db-dir=$DB_DIN
Failed to process planet file

Update repeats in loop without sleep

After initialization using image version 0.7.56.8 (also tried 0.7.55.9) updating process repeats in loop without sleep. Initialization configuration:

docker run \
  -e OVERPASS_META=no \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/monaco-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/monaco/minute/ \
  -e OVERPASS_RULES_LOAD=10 \
  -e OVERPASS_UPDATE_SLEEP=300 \
  -v /mnt/OSM_data/overpass_monaco/:/db \
  -p 12345:80 \
  -i -t \
  --name overpass_monaco wiktorn/overpass-api:0.7.56.8

Also tried with OVERPASS_META=yes and without OVERPASS_RULES_LOAD or OVERPASS_UPDATE_SLEEP variables.
Logs output looks like:

log

There are still some updates remaining
2020-12-28 08:37:57 INFO: Using replication server at http://download.openstreetmap.fr/replication/europe/monaco/minute/
2020-12-28 08:37:57 DEBUG: Using given sequence ID 4346024
2020-12-28 08:37:57 DEBUG: Starting download at ID 4346025 (max 100 MB)
2020-12-28 08:37:57 DEBUG: Downloaded change 4346025. (102400 kB available in download buffer)
2020-12-28 08:37:57 DEBUG: Downloaded change 4346026. (102400 kB available in download buffer)
2020-12-28 08:37:57 DEBUG: Downloaded change 4346027. (102400 kB available in download buffer)
Empty version, skipping file
<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="libosmium/2.15.6">
</osm>
There are still some updates remaining
2020-12-28 08:37:57 INFO: Using replication server at http://download.openstreetmap.fr/replication/europe/monaco/minute/
2020-12-28 08:37:57 DEBUG: Using given sequence ID 4346024
2020-12-28 08:37:57 DEBUG: Starting download at ID 4346025 (max 100 MB)
2020-12-28 08:37:57 DEBUG: Downloaded change 4346025. (102400 kB available in download buffer)
2020-12-28 08:37:57 DEBUG: Downloaded change 4346026. (102400 kB available in download buffer)
2020-12-28 08:37:57 DEBUG: Downloaded change 4346027. (102400 kB available in download buffer)
Empty version, skipping file
<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="libosmium/2.15.6">
</osm>
There are still some updates remaining
2020-12-28 08:37:57 INFO: Using replication server at http://download.openstreetmap.fr/replication/europe/monaco/minute/
2020-12-28 08:37:57 DEBUG: Using given sequence ID 4346024
2020-12-28 08:37:57 DEBUG: Starting download at ID 4346025 (max 100 MB)
2020-12-28 08:37:57 DEBUG: Downloaded change 4346025. (102400 kB available in download buffer)
2020-12-28 08:37:58 DEBUG: Downloaded change 4346026. (102400 kB available in download buffer)
2020-12-28 08:37:58 DEBUG: Downloaded change 4346027. (102400 kB available in download buffer)
Empty version, skipping file
<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="libosmium/2.15.6">
</osm>
There are still some updates remaining

Though version latest which I pulled April 2020 seems to be working fine (it logs Overpass API 0.7.56.1003 though image wiktorn/overpass-api:0.7.56.1003 still loops without sleep). Started container using latest version from April 2020 sleeps after update showing appropriate message:

log

There are still some updates remaining
INFO: Using replication server at http://download.openstreetmap.fr/replication/europe/monaco/minute/
DEBUG: Using given sequence ID 4348922
DEBUG: Starting download at ID 4348923 (max 100 MB)
Empty version, skipping file
<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="libosmium/2.15.4">
</osm>
Update finished with status code: 3
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.56.1003 429967a9">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2020-12-30T01:00:55Z" areas="2020-12-30T01:00:55Z"/>


</osm>
It took 2 to run the loop. Desired load is: 1%. Sleeping: 198
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.56.1003 429967a9">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2020-12-30T01:00:55Z" areas="2020-12-30T01:00:55Z"/>


</osm>
It took 2 to run the loop. Desired load is: 1%. Sleeping: 198
INFO: Using replication server at http://download.openstreetmap.fr/replication/europe/monaco/minute/
DEBUG: Using given sequence ID 4348922
DEBUG: Starting download at ID 4348923 (max 100 MB)

Is it possible to clip diffs according to a polygon?

Hello,

We are interested only for a small region of central europe. We can extract it from planet file and use it for OVERPASS_PLANET_URL. Is it possible to apply our clipping polygon from GeoJSON file to minutely diffs?

404 Not Found errors for retrieving Overpass data

I am running the clone example from the README. The docker image starts successfully, but after downloading data, I have many 404 Not Found errors for retrieving from the Overpass server.

kmehta@kmehta-desktop:~$ sudo docker run   -e OVERPASS_META=yes   -e OVERPASS_MODE=clone   -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/   -v /big/docker/overpass_clone_db/:/db   -p 12346:80   -i -t   --name overpass_world   wiktorn/overpass-api
No database directory. Initializing
--2021-01-19 18:46:58--  http://dev.overpass-api.de/api_drolbr//trigger_clone
Resolving dev.overpass-api.de (dev.overpass-api.de)... 95.217.37.171, 2a01:4f9:2b:2fd4::2
Connecting to dev.overpass-api.de (dev.overpass-api.de)|95.217.37.171|:80... connected.
HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable

    The file is already fully retrieved; nothing to do.

--2021-01-19 18:47:01--  https://dev.overpass-api.de/clone//2021-01-15/replicate_id
Resolving dev.overpass-api.de (dev.overpass-api.de)... 95.217.37.171, 2a01:4f9:2b:2fd4::2
Connecting to dev.overpass-api.de (dev.overpass-api.de)|95.217.37.171|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-01-19 18:47:02 ERROR 404: Not Found.


Fetching nodes.bin
--2021-01-19 18:47:02--  https://dev.overpass-api.de/clone//2021-01-15/nodes.bin
Resolving dev.overpass-api.de (dev.overpass-api.de)... 95.217.37.171, 2a01:4f9:2b:2fd4::2
Connecting to dev.overpass-api.de (dev.overpass-api.de)|95.217.37.171|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-01-19 18:47:02 ERROR 404: Not Found.

First install

Hi, please,
i just try to install an image with command in README

sudo docker run \
  -e OVERPASS_META=no \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -v /big/docker/overpass_clone_db/:/db \
  -p 8083:80 \
  -i -t \
  --name overpass_world \
  wiktorn/overpass-api

but i see only this:

2019-05-24 10:55:05,351 CRIT Supervisor running as root (no user in config file)
2019-05-24 10:55:05,355 INFO supervisord started with pid 1
2019-05-24 10:55:06,359 INFO spawned: 'overpass_dispatch' with pid 9
2019-05-24 10:55:06,361 INFO spawned: 'nginx' with pid 10
2019-05-24 10:55:06,364 INFO spawned: 'fcgiwrap' with pid 11
2019-05-24 10:55:06,367 INFO spawned: 'update_overpass' with pid 12
2019-05-24 10:55:06,370 INFO spawned: 'dispatcher_areas' with pid 15
2019-05-24 10:55:06,372 INFO spawned: 'areas_rules' with pid 16
2019-05-24 10:55:06,376 INFO exited: overpass_dispatch (exit status 0; not expected)
2019-05-24 10:55:07,561 INFO spawned: 'overpass_dispatch' with pid 43
2019-05-24 10:55:07,562 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-24 10:55:07,562 INFO success: fcgiwrap entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-24 10:55:07,562 INFO success: update_overpass entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-24 10:55:07,562 INFO success: dispatcher_areas entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-24 10:55:07,562 INFO success: areas_rules entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-24 10:55:07,574 INFO exited: overpass_dispatch (exit status 0; not expected)
2019-05-24 10:55:10,411 INFO spawned: 'overpass_dispatch' with pid 54
2019-05-24 10:55:10,424 INFO exited: overpass_dispatch (exit status 0; not expected)
2019-05-24 10:55:14,426 INFO spawned: 'overpass_dispatch' with pid 64
2019-05-24 10:55:14,439 INFO exited: overpass_dispatch (exit status 0; not expected)
2019-05-24 10:55:15,431 INFO gave up: overpass_dispatch entered FATAL state, too many start retries too quickly

Should i wait or what is wrong? /big/docker/ directory change only changes.log but no data downloaded.

In changes.log i is:


Traceback (most recent call last):
  File "/app/venv/bin/pyosmium-get-changes", line 243, in <module>
    exit(main(sys.argv[1:]))
  File "/app/venv/bin/pyosmium-get-changes", line 186, in main
    with open(options.seq_file, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/db/replicate_id'
/app/bin/update_database --compression-method= --map-compression-method= --flush-size= --db-dir=/db/db
cat: /db/diffs/changes.osm: No such file or directory
For --compression-method, please use "no", "gz", or "lz4" as value.
For --map-compression-method, please use "no", "gz", or "lz4" as value.
Usage: /app/bin/update_database [--db-dir=DIR] [--version=VER] [--meta|--keep-attic] [--flush_size=FLUSH_SIZE] [--compression-method=(no|gz|lz4)] [--map-compression-method=(no|gz|lz4)]

http.client.IncompleteRead

The get request in python is not working. I am hosting on a gcloud computing instance. With Postman and Curl the request is working, but not with python and the request lib.

Docker configuration:



docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -e OVERPASS_RATE_LIMIT=1 \
  -e OVERPASS_SPACE=8589934592 \
  -v /big/docker/overpass_clone_db/:/db \
  -p 12346:80 \
  -i -t \
  --name overpass_world \
  wiktorn/overpass-api

docker start overpass_world

Working:
curl --location -g --request GET 'http://35.242.198.27:12346/api/interpreter?data=[timeout:3600][maxsize:1073741824][out:json];node(around:5000,51.408159,7.2989356,51.4030306,7.3490601,51.4364065,7.3528927,51.4466813,7.3238974,51.408159,7.2989356)[name];out;%0A'

Not working:


url = "http://35.242.198.27:12346/api/interpreter?data=[timeout:3600][maxsize:1073741824][out:json];node(around:5000,51.408159,7.2989356,51.4030306,7.3490601,51.4364065,7.3528927,51.4466813,7.3238974,51.408159,7.2989356)[name];out;"

response = requests.request("GET", url)
#
print(response.text)`


Produces the following error:

Traceback (most recent call last):
  File "/home/julian/develop/dielage/backend/real-estate-integrator/venv/lib/python3.8/site-packages/urllib3/response.py", line 436, in _error_catcher
    yield
  File "/home/julian/develop/dielage/backend/real-estate-integrator/venv/lib/python3.8/site-packages/urllib3/response.py", line 766, in read_chunked
    chunk = self._handle_chunk(amt)
  File "/home/julian/develop/dielage/backend/real-estate-integrator/venv/lib/python3.8/site-packages/urllib3/response.py", line 710, in _handle_chunk
    value = self._fp._safe_read(amt)
  File "/usr/lib/python3.8/http/client.py", line 614, in _safe_read
    raise IncompleteRead(data, amt-len(data))
http.client.IncompleteRead: IncompleteRead(5461 bytes read, 4779 more expected)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/julian/develop/dielage/backend/real-estate-integrator/venv/lib/python3.8/site-packages/requests/models.py", line 751, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/home/julian/develop/dielage/backend/real-estate-integrator/venv/lib/python3.8/site-packages/urllib3/response.py", line 571, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/home/julian/develop/dielage/backend/real-estate-integrator/venv/lib/python3.8/site-packages/urllib3/response.py", line 792, in read_chunked
    self._original_response.close()
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/julian/develop/dielage/backend/real-estate-integrator/venv/lib/python3.8/site-packages/urllib3/response.py", line 454, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(5461 bytes read, 4779 more expected)', IncompleteRead(5461 bytes read, 4779 more expected))

During handling of the above exception, another exception occurred:

Following the build instructions for the clone results in a 403 error on access

After following the middle instruction to clone a new world using the VM everything appears to work, but on accessing the port we get a 403 error from the Nginx running inside the docker instance. It looks like the files have downloaded correctly and docker does seem to have permissions to access the folder that it has downloaded the database files to.

If this happened to someone else, am I missing something obvious?

Setting OVERPASS_META to "no" results in startup error.

This might be a regression from be98c86.

Setting the environment variable OVERPASS_META to no results in the following error message when trying to boot the dispatcher:

File /db/db/nodes_meta.bin present. Please use parameter --meta
File /db/db/ways_meta.bin present. Please use parameter --meta
File /db/db/relations_meta.bin present. Please use parameter --meta
File /db/db/user_data.bin present. Please use parameter --meta
File /db/db/user_indices.bin present. Please use parameter --meta

I believe this issue is here: https://github.com/wiktorn/Overpass-API/blob/master/docker-entrypoint.sh#L45

According to the help message, and this comment drolbr/Overpass-API#446 (comment) we should either omit or include the --meta flag.

Usage:  /app/bin/init_osm3s.sh  Planet_File  Database_Dir  Executable_Dir  [--meta]
        where
    Planet_File is the filename and path of the compressed planet file, including .bz2,
    Database_Dir is the directory the database should go into, and
    Executable_Dir is the directory that contains the executable update_database.
    Add --meta in the end if you want to use meta data.

Inconsistent behavior using docker-compose

Issue:

When trying to access /api(http://localhost:12345) from the host machine, not a single page loads with the response:

localhost didnโ€™t send any data.
ERR_EMPTY_RESPONSE

Steps to replicate:
Chrome: Version 87.0.4280.88 (Official Build) (64-bit)
OS: Windows, Docker for Windows 3.0.0 (50684), Engine: 20.10.0, WSL2 (20.04.1 LTS)
Overpassversion: tested both on 0.7.56 and latest tag

The following docker command works as described in the documentation works:

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/austria-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=https://download.openstreetmap.fr/replication/europe/austria/minute/ \
  -e OVERPASS_RULES_LOAD=10 \
  -v /home/neatnerd/backend/data/overpass/:/db \
  -p 12345:80 \
  -i -t \
  --name overpass_austria wiktorn/overpass-api

The following docker-compose file does not work:

version: "3"

services:
  overpass:
    image: wiktorn/overpass-api
    ports: 
      - "12345:8080"
    volumes:
      - ./data/overpass/:/db
    environment: 
      - OVERPASS_META=yes
      - OVERPASS_MODE=init
      - OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/austria-latest.osm.bz2
      - OVERPASS_DIFF_URL=https://download.openstreetmap.fr/replication/europe/austria/minute/
      - OVERPASS_RULES_LOAD=10

I have tried to switch off the firewall and anti-virus. The most strange behavior is that I believe the page pops up on startup and then goes down, but this might be browser cache.

Logs

Starting supervisord process

2020-12-25 08:49:30,870 CRIT Supervisor running as root (no user in config file)
2020-12-25 08:49:30,872 INFO supervisord started with pid 1
2020-12-25 08:49:31,875 INFO spawned: 'overpass_dispatch' with pid 10
2020-12-25 08:49:31,877 INFO spawned: 'nginx' with pid 11
2020-12-25 08:49:31,880 INFO spawned: 'fcgiwrap' with pid 12
2020-12-25 08:49:31,884 INFO spawned: 'update_overpass' with pid 13
2020-12-25 08:49:31,889 INFO spawned: 'dispatcher_areas' with pid 14
2020-12-25 08:49:31,892 INFO spawned: 'areas_rules' with pid 15
/app/bin /
/db/diffs/changes.osm exists. Trying to apply again.
XML parsing error at line 3, column 0: no element found
2020-12-25 08:49:32,929 INFO success: overpass_dispatch entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-12-25 08:49:32,929 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-12-25 08:49:32,929 INFO success: fcgiwrap entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-12-25 08:49:32,929 INFO success: update_overpass entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-12-25 08:49:32,929 INFO success: dispatcher_areas entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-12-25 08:49:32,929 INFO success: areas_rules entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.56.3 eb200aeb">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2020-12-23T23:31:53Z" areas="2020-12-23T23:31:53Z"/>
After 0h0m15s: in "recurse", part 0, on line 26. Stack: 0 of 0 518 of 0 0 of 0
After 0h0m30s: in "recurse", part 0, on line 27. Stack: 0 of 0 1449 of 0 0 of 0
After 0h0m45s: in "make-area", part 0, on line 29. Stack: 0 of 0 2394 of 0
After 0h1m0s: in "recurse", part 0, on line 27. Stack: 0 of 0 3255 of 0 0 of 0
/db/diffs/changes.osm exists. Trying to apply again.

Desired behavior

Docker-compose settings shall result in the same outcome as running individual docker commands

Runtime error: The dispatcher (i.e. the database management system) is turned off. Kubernetes

I was able to run the container for australia region and the docker container restarted after downloading entire db and the pod showed 1/1 on kubernetes . However when I check the logs and query for the api . I get strange error as -

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
  <meta http-equiv="content-type" content="text/html; charset=utf-8" lang="en"/>
  <title>OSM3S Response</title>
</head>
<body>

<p>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</p>
<p><strong style="color:#FF0000">Error</strong>: runtime error: The dispatcher (i.e. the database management system) is turned off. </p>

</body>
</html>

Traceback (most recent call last):
  File "data_task__by_nikhil.py", line 140, in <module>
    data_frame = extract_raw_data_from_OSM(query)
  File "data_task__by_nikhil.py", line 118, in extract_raw_data_from_OSM
    json_data = response.json()
  File "/usr/lib/python3.8/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Below are the logs from the pod

It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
Update finished with status code: 3
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
10.240.0.115 - - [14/Nov/2020:13:22:49 +0000] "GET /api/status HTTP/1.1" 200 156 "-" "kube-probe/1.19"
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
10.240.0.115 - - [14/Nov/2020:13:22:59 +0000] "GET /api/status HTTP/1.1" 200 156 "-" "kube-probe/1.19"
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 1 to run the loop. Desired load is: 10%. Sleeping: 9
10.240.0.115 - - [14/Nov/2020:13:23:09 +0000] "GET /api/status HTTP/1.1" 200 156 "-" "kube-probe/1.19"
10.240.0.115 - - [14/Nov/2020:13:23:12 +0000] "GET /api/interpreter?data=%5Bout%3Ajson%5D%5Btimeout%3A50%5D%3B%28way%5B%22highway%22%5D%28around%3A50%2C-34.06221%2C150.82524%29%5Bmaxspeed%5D%3B%29%3Bout+body%3B HTTP/1.1" 504 635 "-" "python-requests/2.24.0"
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
10.240.0.115 - - [14/Nov/2020:13:23:19 +0000] "GET /api/status HTTP/1.1" 200 155 "-" "kube-probe/1.19"
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3

Can you please guide how and why is this happening and how to resolve this ?

Failed to process planet file

Hi,

I'm running into a problem running the docker image on a windows machine. I'm trying to run it for New Zealand using the geofabric site:

docker run \ -e OVERPASS_META=yes \ -e OVERPASS_MODE=init \ -e OVERPASS_PLANET_URL=http://download.geofabrik.de/australia-oceania/new-zealand-latest.osm.bz2 \ -e OVERPASS_DIFF_URL=http://download.geofabrik.de/australia-oceania/new-zealand-updates/ \ -e OVERPASS_RULES_LOAD=10 \ -e OVERPASS_UPDATE_SLEEP=3600 \ -v E:\ecan\deployments\overpass\NZ\:/db \ -p 12345:80 \ -i -t \ --name overpass_nz wiktorn/overpass-api

Gives me the error after initially processing the planet file:

No database directory. Initializing % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 373M 100 373M 0 0 7075k 0 0:00:54 0:00:54 --:--:-- 6675k [======================================================================] 100% Reading XML file ... elapsed node 2945583286. /app/bin/init_osm3s.sh: line 44: 16 Broken pipe bunzip2 < $PLANET_FILE 17 Killed | $EXEC_DIR/bin/update_database --db-dir=$DB_DIR/ $META $COMPRESSION Failed to process planet file

My guess is because it's a windows mount where the database sits rather than Linux? The sh script doesn't account for this?

Thanks!

Example env vars incorrect?

I'm trying to run the example using the below, slightly modified code

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/poland-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/poland/minute/ \
  -e OVERPASS_PLANET_SEQUENCE_ID=2346486 \
  -p 12345:80 \
  -i -t \
  --name overpass_poland wiktorn/overpass-api:0.7.55

But it throws an error with the supervisord.conf

Error: Format string 'nice /app/bin/rules_loop.sh /db/db %(ENV_OVERPASS_RULES_LOAD)s' for 'program:areas_rules.command' contains names ('ENV_OVERPASS_RULES_LOAD') which cannot be expanded. Available names: ENV_HOME, ENV_HOSTNAME, ENV_NGINX_VERSION, ENV_NJS_VERSION, ENV_OVERPASS_DIFF_URL, ENV_OVERPASS_META, ENV_OVERPASS_MODE, ENV_OVERPASS_PLANET_SEQUENCE_ID, ENV_OVERPASS_PLANET_URL, ENV_PATH, ENV_PWD, ENV_SHLVL, ENV_TERM, group_name, here, host_node_name, process_num, program_name in section 'program:areas_rules' (file: '/etc/supervisor/conf.d/supervisord.conf')
For help, use /usr/bin/supervisord -h

Have the env var names changed?

Sometime update got stuck

Hi,

Sometime, the update process got stuck with this message (repeated durings hours ....) :

INFO: Using replication server at http://download.openstreetmap.fr/replication/europe/france/minute/
DEBUG: Using given sequence ID 4039576
DEBUG: Starting download at ID 4039577 (max 100 MB)
DEBUG: Loading state info None failed with: <urlopen error [Errno 99] Cannot assign requested address>
Empty version, skipping file
<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="libosmium/2.15.4">
</osm>
Update finished with status code: 3

it seems that pyosmium-get-changes got an error at certain point ?
but why ?

Detailed steps for using full attic data?

Has anyone successfully setup an Overpass API server with full (same as Overpass DE i.e. from 2012) attic data (for a certain region / country)? I noticed an earlier issue #5 touching this topic, with reference to instructions in Overpass wiki.

However after reading those instructions as well as some further references around the question (e.g. another wiki entry) I was not sure whether this is feasible and how much effort it would require in terms of learning all the details of the process, obtaining all data files etc.

Has someone done this and if so, could provide simple "for dummies" instructions? Or at least tell what kind of effort to expect to make it work?

(For my use case I'm actually interested mostly on historical "attic" data rather than applying new diffs to stay up-to-date...)

502 response error - runtime error

Hi again,

Sorry for the constant issues.

I've run it with the following docker command (and subsequently started it):

docker run \ -e OVERPASS_META=yes \ -e OVERPASS_MODE=init \ -e OVERPASS_PLANET_URL=http://download.geofabrik.de/australia-oceania/new-zealand-latest.osm.bz2 \ -e OVERPASS_DIFF_URL=http://download.geofabrik.de/australia-oceania/new-zealand-updates/ \ -e OVERPASS_RULES_LOAD=10 \ -e OVERPASS_UPDATE_SLEEP=3600 \ -v E:\ecan\deployments\overpass\NZ\:/db \ -p 12345:80 \ -i -t \ --name overpass_nz wiktorn/overpass-api

Whenever I send a request to the overpass api I get a 502 response error with the following content:

<?xml version="1.0" encoding="UTF-8"?>\n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"\n "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">\n<head>\n <meta http-equiv="content-type" content="text/html; charset=utf-8" lang="en"/>\n <title>OSM3S Response</title>\n</head>\n<body>\n\n<p>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</p>\n<p><strong style="color:#FF0000">Error</strong>: runtime error: The dispatcher (i.e. the database management system) is turned off. </p>\n\n</body>\n</html>

It says the dispatcher is turned off...I am running the container...

Thanks again.

Unable to initiate docker

Hi,
Tried to initiate the docker like in the example.
I think the example referring to ./docker-entrypoint.sh? Didn't find any Docker file to run.
Also when did that, the script prompt:
image

Help would be very much appreciated!

very limited max connexion ? only 4 per seconds ?

i have up to 20 req/s to my overpass server using you docker image, and it quickly send HTTP 502 error with this message :

[error] 19#19: *717 connect() to unix:/nginx/fcgiwrap.socket failed (11: Resource temporarily unavailable) while connecting to upstream

I try to change a lot of nginx configuration, but none is working.

To test it, i have a Gatling script that send 8 request / seconds.
sample request :
[timeout:180][out:json];way(48.4655000,7.7280000,48.4656000,7.7281990)[highway];(._;>;);out;

Gatling result show that the serveur only reply 4 time / seconds. never more.

overpass log rotate setting inside docker image ?

overpass spawn a lot of logs in changes.log and transactions.log. (rules_loop.log also but it's a smaller size).

it is possible to use logrotate outside of the docker image to limit the size of thoses files. BUT it would be nice if the log rotate things is package inside the docker image .

do you think it's a good idea, or logrotate outside is the only good way to do it ?

Cloning uses older dataset of two available

I am trying to install the latest image using this command```

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -v /Volumes/Test/Docker/overpass_clone_db/:/db \
  -p 12346:80 \
  -i -t \
  --name overpass_world \
  wiktorn/overpass-api

There are two images on https://dev.overpass-api.de/clone/ at the moment:


ย  | Name | Last modified | Size | Description
ย  | 2020-06-11/ | 2020-06-11 09:01 | - | ย 
ย  | 2020-06-12/ | 2020-06-12 05:44 | - | ย 

When I start the image installation, it starts to download the old image (2020-06-11) though instead of the newer one. I need the newer one as download takes more than 24 hours on my internet connection and the old image will disappear within that time.

Trying to install this docker image on digital ocean with attached volume [question]

Thanks for creating this docker image, works great with smaller datasets. However we're now trying to set this up on digital ocean with an attached volume mounted on host machine /mnt/... However the install script doesnt seem to pick it up when I run it like below:

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/minute/ \
  -e OVERPASS_PLANET_SEQUENCE_ID=2946200 \
  -e OVERPASS_RULES_LOAD=10 \
  -v /big/docker/overpass_db/:/mnt/volume_nyc3_01/db \
  -p 12345:80 \
  -i -t \
  --name overpass_europe wiktorn/overpass-api

Any idea what we're doing wrong and how to modify it so the DB is stored on the mounted volume?

Thanks!

Memory issues when initializing container

Hi,

I tried to initialize a container using a tarball of whole Europe (~32GB) on my machine having 8GB of memory and an SSD disk.

I expexted it would take a while but not so much... it's running since yesterday. After downloading the snapshot, my linux box started swapping and all I see from the output of the container is a buch of lines like:

  • Reading XML file.... elapsed way 267461502. Flushing the database...... done. with the integer increasing
  • and Reorganizing the database.... done.

Do you recommend me killing the process and trying something else? Or should I wait? I'm currently on vacation so i dont have any problem letting the machine run for another day.

Is there a way to predict more or less how much time is going to require until the end?

Is there a way to tweak the init run to avoid running into swapping issues?

Any suggestion?

Thanks in advance

docker start bug

hello,
version: latest
desc: when i use docker start the overpass server , tips :

Reading XML file .../app/bin/init_osm3s.sh: line 44: 10 Broken pipe bunzip2 < $PLANET_FILE
11 Killed | $EXEC_DIR/bin/update_database --db-dir=$DB_DIR/ $META $COMPRESSION
how to fix it;
thanks!

How to use the predownloaded osm.bz file when initializing docker?

It is not really an issue, I would just like to know on how can we use local file as source for OSM_PLANET_URL parameter. I have read from the update that file:/// scheme is now supported in OSM_PLANET_URL parameter. I tried several ways but have encountered several errors like:

1.) OVERPASS_PLANET_URL=file://C:/file/path/to/sample.osm.bz2
error: curl: (3) Valid host name with slash missing in URL

2.) OVERPASS_PLANET_URL=file:///C:/file/path/to/sample.osm.bz2
error: curl: (37) Couldn't open file /C:/file/path/to/sample.osm.bz2

3.) OVERPASS_PLANET_URL='file:///C:/file/path/to/sample.osm.bz2'
error: curl: (1) Protocol "'file" not supported or disabled in libcurl

I hope you can provide some assistance. Thank you

Area search doesn't seem to work

After a 24 hours parsing of the Europe dataset from Geofabrik I have the overpass api succesfully running inside the docker container.
Simple overpass queries seem to work just fine but for some reason area limiting doesn't seem to work, no error, just no results. Below query works fine on the public overpass-turbo server but not on my own instance.

{{geocodeArea:Germany}}->.country;
rel(area.country)["boundary"="administrative"]["name"~"Main"][admin_level=10];
out geom;

area["ISO3166-1"="DE"][admin_level=2];
rel["boundary"="administrative"]["name"~"Mainz"][admin_level=6](area);
out geom;

When I remove the area restriction it works but it returns all matches inside Europe.

Any idea?

Option for (currently static) nginx timeouts?

First of all @wiktorn , thank you for this Docker image! Even for someone who is just starting to learn Docker it was very easy to setup an overpass server with the help of this image.

While I was running some tests today querying my docker overpass instance through wget, I noticed that the connection to the server was reset roughly every 16 minutes. As my query needs to run longer (this is the main reason for my private overpass instance), wget then started a second query which resulted in the same two queries being executed on the server at the same time with an offset of 16 mins.

The culprit in this case is the fastcgi_read_timeout 1000s; directive in nginx.conf.

So my question/feature request is to pass that value as an argument to docker run if possible. Alternatively one could think about a higher default value (like 24 hours or so).

Thanks and best regards!

[Edit] Changed keepalive_timeout to fastcgi_read_timeout.

Deploy to Kubernetes?

Hi

I'm trying to deploy this to (google) kubernetes but it's not working. Any ideas on what I need to change in this yaml?

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: overpass
spec:
  template:
    metadata:
      labels:
        name: overpass
    spec:
      containers:
        - name: overpass
          image: wiktorn/overpass-api
          volumeMounts:
          - mountPath: /db
            name: overpass-volume
          readinessProbe:
            httpGet:
              path: /interpreter
              port: 80
          ports:
            - name: overpass
              containerPort: 80
          env:
          - name: OVERPASS_META
            value: "no"
          - name: OVERPASS_MODE
            value: "init"
          - name: OVERPASS_PLANET_URL
            value: "http://download.geofabrik.de/australia-oceania/australia-latest.osm.bz2"
          - name: OVERPASS_DIFF_URL
            value: "http://download.openstreetmap.fr/replication/oceania/australia/minute/"
          - name: OVERPASS_PLANET_SEQUENCE_ID
            value: "3325745"
          - name: OVERPASS_RULES_LOAD
            value: "10"
      volumes:
      - name: overpass-volume
        emptyDir: {}

Failed to build docker

Trying to build a latest docker image locally with following docker-compose:

overpass:
    container_name: overpass
    build: 0.7.55.7/.
    cpu_shares: 2048
    mem_limit: 4096M
    environment:
        - OVERPASS_META=yes
        - OVERPASS_MODE=init
        - OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/monaco-latest.osm.bz2
        - OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/monaco/minute/
        - OVERPASS_RULES_LOAD=10
    ports:
        - 12345:80
    volumes:
        - ~/dump/db/:/db:rw

getting this:

/bin/bash ./libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -O2 -MT template_db/dispatcher_client.lo -MD -MP -MF $depbase.Tpo -c -o template_db/dispatcher_client.lo template_db/dispatcher_client.cc &&
mv -f $depbase.Tpo $depbase.Plo
libtool: compile: g++ -DHAVE_CONFIG_H -I. -O2 -MT template_db/dispatcher_client.lo -MD -MP -MF template_db/.deps/dispatcher_client.Tpo -c template_db/dispatcher_client.cc -fPIC -DPIC -o template_db/.libs/dispatcher_client.o
libtool: compile: g++ -DHAVE_CONFIG_H -I. -O2 -MT template_db/dispatcher_client.lo -MD -MP -MF template_db/.deps/dispatcher_client.Tpo -c template_db/dispatcher_client.cc -o template_db/dispatcher_client.o >/dev/null 2>&1
/bin/bash ./libtool --tag=CXX --mode=link g++ -O2 -o libexpatwrapper.la expat/expat_justparse_interface.lo -lexpat -lrt -lexpat
libtool: error: 'expat/expat_justparse_interface.lo' is not a valid libtool object
Makefile:1102: recipe for target 'libexpatwrapper.la' failed
make[1]: *** [libexpatwrapper.la] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory '/app/src'
Makefile:1873: recipe for target 'install-recursive' failed
make: *** [install-recursive] Error 1
ERROR: Service 'overpass' failed to build: The command '/bin/sh -c mkdir -p /app/src && cd /app/src && tar -x -z --strip-components 1 -f ../src.tar.gz && autoscan && aclocal && autoheader && libtoolize && automake --add-missing && autoconf && CXXFLAGS='-O2' CFLAGS='-O2' ./configure --prefix=/app --enable-lz4 && make -j $(grep -c ^processor /proc/cpuinfo) dist install clean && mkdir -p /db/diffs /app/etc && cp -r /app/src/rules /app/etc/rules && rm -rf /app/src /app/src.tar.gz' returned a non-zero code: 2

Dispatcher Client timeout

I successfully created an overpass instance with this Docker image about a month ago. When I was testing different parameters/configurations the past weeks, I deleted the working container and volume and created new containers. But since a few weeks they all fail to work.

The creation of the overpass instance seems to go through fine, nginx is responding to queries, but all queries time out and nginx returns a 504. Creating the overpass_monaco example does work. Using this or similar commands cloning the planet reproduce the timeout issue:

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -e OVERPASS_MAX_TIMEOUT=14400s \
  -e OVERPASS_SPACE=16777216 \
  -p 12345:80 \
  -d \
  --mount source=overpass,target=/db \
  --restart unless-stopped \
  --name overpass \
  wiktorn/overpass-api

The response from nginx is always

<p>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</p>
<p><strong style="color:#FF0000">Error</strong>: runtime error: open64: 0 Success /osm3s_v0.7.55_osm_base Dispatcher_Client::request_read_and_idx::timeout. The server is probably too busy to handle your request. </p>

"Too busy" cannot be the real reason as I am the only user running simple queries to test.

http://myhost.com:12345/api/status returns

Connected as: 1559890362
Current time: 2020-07-16T15:51:08Z
Rate limit: 0
Currently running queries (pid, space limit, time limit, start time):

so all good here.

changes.log looks OK. Updates are being made.
transactions.log looks a bit suspicious:

2020-07-16 15:17:20 [16] Dispatcher version 0.7.56.3 eb200aebdae27bd8635c5829a2bdfcdb06d0ab5c just started.
2020-07-16 15:17:20 [15] Dispatcher version 0.7.56.3 eb200aebdae27bd8635c5829a2bdfcdb06d0ab5c just started.
2020-07-16 15:17:20 [33] request_read_and_idx() start
2020-07-16 15:17:21 [16] hangup of process 27.
2020-07-16 15:17:21 [16] waited idle for 8 cycles.
2020-07-16 15:17:21 [16] waited idle for 8 cycles.
[...]
2020-07-16 15:17:56 [16] waited idle for 8 cycles.
2020-07-16 15:17:57 [33] Dispatcher_Client::request_read_and_idx::timeout /osm3s_v0.7.55_osm_base 0 Success 
2020-07-16 15:17:57 [16] waited idle for 8 cycles.
2020-07-16 15:17:57 [16] hangup of process 33.
2020-07-16 15:18:14 [818] write_start() start version='2020-07-16T06:23:01Z'
2020-07-16 15:18:14 [16] waited idle for 173 cycles.
2020-07-16 15:18:14 [16] write_start of process 818. Considered as reading:.
2020-07-16 15:18:14 [818] write_start() end
2020-07-16 15:34:20 [819] request_read_and_idx() start
2020-07-16 15:34:20 [16] waited idle for 9625 cycles.
[...]
2020-07-16 15:34:56 [16] waited idle for 8 cycles.
2020-07-16 15:34:56 [819] Dispatcher_Client::request_read_and_idx::timeout /osm3s_v0.7.55_osm_base 0 Success 92.250.9.186
2020-07-16 15:34:56 [16] waited idle for 8 cycles.
2020-07-16 15:34:56 [16] hangup of process 819.
2020-07-16 15:40:34 [832] request_read_and_idx() start
2020-07-16 15:40:34 [16] waited idle for 3365 cycles.
[...]
2020-07-16 15:48:28 [16] waited idle for 8 cycles.
2020-07-16 15:48:28 [833] Dispatcher_Client::request_read_and_idx::timeout /osm3s_v0.7.55_osm_base 0 Success 92.250.9.186
2020-07-16 15:48:28 [16] waited idle for 8 cycles.
2020-07-16 15:48:28 [16] hangup of process 833.
2020-07-16 15:50:14 [16] waited idle for 1057 cycles.
2020-07-16 15:50:14 [16] query_my_status by process 836.
2020-07-16 15:50:14 [16] waited idle for 1 cycles.
2020-07-16 15:50:14 [16] hangup of process 836.
2020-07-16 15:51:08 [16] waited idle for 546 cycles.
2020-07-16 15:51:08 [16] query_my_status by process 839.
2020-07-16 15:51:08 [16] waited idle for 1 cycles.
2020-07-16 15:51:08 [16] hangup of process 839.
2020-07-16 15:51:44 [818] write_commit() start 149937 1838062 420189 1134147 270038
2020-07-16 15:51:44 [16] waited idle for 357 cycles.
2020-07-16 15:51:44 [16] write_commit of process 818.
2020-07-16 15:51:44 [818] write_commit() end
2020-07-16 15:51:44 [16] hangup of process 818.
2020-07-16 15:52:22 [1267] write_start() start version='2020-07-16T09:50:01Z'
2020-07-16 15:52:22 [16] waited idle for 388 cycles.
2020-07-16 15:52:22 [16] write_start of process 1267. Considered as reading:.
2020-07-16 15:52:22 [1267] write_start() end

I googled all over the place but cannot find a clue.

Thanks for any pointers!

No (or empty) response form server - Request returns Error Code 403

Hey there,

if been trying to set up the overpass API for Germany. Therefore I used the provided script and modified it:
docker run \ -e OVERPASS_META=yes \ -e OVERPASS_MODE=init \ -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/germany-latest.osm.bz2 \ -e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/germany/minute/ \ -e OVERPASS_RULES_LOAD=10 \ -v /big/docker/overpass_db/:/db \ -p 12345:80 \ -i -t \ --name overpass_germany wiktorn/overpass-api

it took a while but now everything is set up. I did the "docker start overpass_germany" comand and it booted. When i am now querying the server with "http;//<server_ip>:12345/api/ I do get error code 404.

Any ideas how I can fix it?

overpass.errors.ServerRuntimeError: runtime error: Query ran out of memory

Hey I am getting the following error when quering the overpass API:

overpass.errors.ServerRuntimeError: runtime error: Query ran out of memory in "query" at line 1. It would need at least 517 MB of RAM to continue.

Since the server has 16GB of Ram and only 20% is in use - is there any way to give Overpass more memory ressources?

Greetings

Area query results different from Overpass Turbo

Here's my query to collect all of the States within the US:

[out:json];
area['admin_level'='2']['name'='United States'];
(relation['admin_level'='4'](area););
out tags;

My Overpass server returns Baja California in the results for this query, even though this state is in Mexico. Overpass Turbo correctly excludes this from the results.

I'm using the wiktorn/overpass-api Docker image that I started it with:

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -v /home/ubuntu/overpass_clone_db/:/db \
  -p 80:80 \
  -i -t \
  --name overpass_world \
  wiktorn/overpass-api

Is there something that I can do to correct these results?
Are there certain places I can look for signs of trouble?

I've also posted this in StackOverflow, GIS StackExchange, and OpenStreetMap Help.

Minutely updater failing and not restarting with container

The minutely downloader seems to be crashing and it doesn't seem like it restarts when the docker instance is restarted.

I've noticed the minutely updater can be unstable in our previous (non-docker) instance so I know that it can be prone to crashing; I've also seen it restart from a previous point inside a docker instance so I'm confident it's configured correctly normally.

Are there any files I should grab off the container that might help in figuring this out? We'll keep it around for debugging purposes to triangulate this issue.

Permission denied

Hi,
First of all thank you for the container!
I have a problem with the container and can't find the cause. I start the container with the following command:
docker run
-e OVERPASS_META=yes
-e OVERPASS_MODE=init
-e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/germany/baden-wuerttemberg/stuttgart-regbez-latest.osm.bz2
-e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/germany/minute/
-e OVERPASS_RULES_LOAD=10
-v overpass:/db/
-p 12345:80
-i -t
--name overpass_bw wiktorn/overpass-api

The data is downloaded and then the container is finished. After the restart I get a permission error in the log:

It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
/app/bin/rules_loop.sh: line 42: ///db/db/rules_loop.log: Permission denied
/app/bin/rules_loop.sh: line 43: ///db/db/rules/areas.osm3s: No such file or directory
/app/bin/rules_loop.sh: line 44: ///db/db/rules_loop.log: Permission denied
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
/app/bin/rules_loop.sh: line 42: ///db/db/rules_loop.log: Permission denied
/app/bin/rules_loop.sh: line 43: ///db/db/rules/areas.osm3s: No such file or directory
/app/bin/rules_loop.sh: line 44: ///db/db/rules_loop.log: Permission denied
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
/app/bin/rules_loop.sh: line 42: ///db/db/rules_loop.log: Permission denied
/app/bin/rules_loop.sh: line 43: ///db/db/rules/areas.osm3s: No such file or directory
/app/bin/rules_loop.sh: line 44: ///db/db/rules_loop.log: Permission denied
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3

Response:
The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.
Error: runtime error: The dispatcher (i.e. the database management system) is turned off.

Is there any indication of what exactly the problem is?

Dummy metadata in Geofabrik public extract downloads.

The example "Overpass instance covering part of the world" uses a geofabrik file where all users have been mapped to a dummy user (user name = "", uid = 0) because of some GDPR requirements. Obviously, this is not ideal. To make this work as before, the extract needs to be downloaded from https://osm-internal.download.geofabrik.de/index.html --> "Login with your OpenStreetMap account "

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/monaco-latest.osm.bz2 \

Running the overpass-api image in a serverless environment

Hello,

I have been trying to see if I can run the image on Google Cloud Run - fully managed.

It seems that docker-entrypoint.sh does the heavy lifting of setting up overpass environment during the first run.

I was curious if this overpass-db setup process could be done as part of docker image building process, such that overpass-db is ready to be used with the image, and the entry point script will only start the nginx server.

This is because Google Cloud Run - fully managed does not provide a mount, and there is no concept of 'first run', 'non-first run' as everything is ephemeral except the image itself.

e.g. If I use the current image, it will attempt to initialize overpass-db by downloading planet file per every request.

Do you have any guidance on how to do that? (or if this cannot be done for some reason?)

Welcome to nginx

Ok, this is probably a really stupid question but I'm pretty new to all this, 'how do I use this'?

I've been using https://overpass-turbo.eu/ but keep hitting rate limiting and query timeouts so decided to give wiktorn/Overpass-API a try. I initialized with the following command:

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -v ${HOME}/overpass:/db \
  -p 12346:80 \
  -i -t \
  --name overpass_world \
  wiktorn/overpass-api

I had to leave the office that evening so suspended my machine, leaving one of the downloads hanging.

When I returned in the morning, I restarted the container, and opened localhost:12346 - I was greeted by the "Welcome to Nginx" page.

The logs looked like this :

2018-06-22 09:28:28,329 CRIT Supervisor running as root (no user in config file)
2018-06-22 09:28:28,334 INFO supervisord started with pid 1                   
2018-06-22 09:28:29,339 INFO spawned: 'overpass_dispatch' with pid 9    
2018-06-22 09:28:29,341 INFO spawned: 'nginx' with pid 10               
2018-06-22 09:28:29,343 INFO spawned: 'fcgiwrap' with pid 11          
2018-06-22 09:28:29,345 INFO spawned: 'apply_diff' with pid 12
2018-06-22 09:28:29,347 INFO spawned: 'fetch_diff' with pid 15
2018-06-22 09:28:29,350 INFO spawned: 'dispatcher_areas' with pid 18           
2018-06-22 09:28:29,354 INFO spawned: 'areas_rules' with pid 19                 
2018-06-22 09:28:29,380 INFO exited: apply_diff (exit status 1; not expected)
2018-06-22 09:28:31,135 INFO success: overpass_dispatch entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-06-22 09:28:31,135 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-06-22 09:28:31,135 INFO success: fcgiwrap entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-06-22 09:28:31,139 INFO spawned: 'apply_diff' with pid 47
2018-06-22 09:28:31,140 INFO success: fetch_diff entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-06-22 09:28:31,140 INFO success: dispatcher_areas entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-06-22 09:28:31,141 INFO success: areas_rules entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-06-22 09:28:32,427 INFO success: apply_diff entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)                                                                      

I entered the container, executed the download_clone command, and allowed it to complete.

docker exec -ti overpass_world bash
/app/bin/download_clone.sh --db-dir=/db/db --source=http://dev.overpass-api.de/api_drolbr/ --meta=yes

I then stopped and started the container, but still just get the "Welcome to Nginx" home page.

I just went to http://overpass-turbo.eu (changing from https to http) - and changed the settings so it would send reqests to http://localhost:12346/api/

Tested it, and tada ! it worked !

So I suggest document how to use (although its obvious to anyone in the community ) and perhaps add my workaround for the failed initial download.

Problem after importing data

I tried to create an overpass instance out of tags wiktorn/overpass-api:latest and wiktorn/overpass-api:0.7.55 but after importing, the container seems to fail.

Here's what I did:

docker volume create overpass
docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/germany/nordrhein-westfalen/muenster-regbez-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -v overpass:/db \
  -p 12345:80 \
  -i -t \
  --name overpass_muenster wiktorn/overpass-api

docker start overpass_muenster

The logs of the overpass_muenster container are:

Reading XML file ... finished reading nodes. Flushing to database ...... done.                                            
Reading XML file ... elapsed way 320277976. Flushing to database ...... done.
Reading XML file ... finished reading ways. Flushing to database ...... done.
Reorganizing the database ... done.
Reading XML file ... finished reading relations. compute_geometry: Way 6654659 used in relation 4531 not found.
compute_geometry: Way 6650667 used in relation 4531 not found.
compute_geometry: Way 6651330 used in relation 4531 not found.
compute_geometry: Way 6651327 used in relation 4531 not found.
compute_geometry: Way 112448606 used in relation 4531 not found.

... many more of these ...

compute_geometry: Way 32731138 used in relation 9332852 not found.
compute_geometry: Way 30016603 used in relation 9332852 not found.
compute_geometry: Way 528727430 used in relation 9332852 not found.
compute_geometry: Way 37560917 used in relation 9332852 not found.
compute_geometry: Way 329748412 used in relation 9332852 not found.
Flushing to database ....... done.
Update complete.
Overpass ready, you can start your container with docker start
2019-03-05 21:42:31,815 CRIT Supervisor running as root (no user in config file)
2019-03-05 21:42:31,820 INFO supervisord started with pid 1
2019-03-05 21:42:32,824 INFO spawned: 'overpass_dispatch' with pid 8
2019-03-05 21:42:32,828 INFO spawned: 'nginx' with pid 9
2019-03-05 21:42:32,835 INFO spawned: 'fcgiwrap' with pid 10
2019-03-05 21:42:32,837 INFO spawned: 'apply_diff' with pid 11
2019-03-05 21:42:32,839 INFO spawned: 'fetch_diff' with pid 12
2019-03-05 21:42:32,841 INFO spawned: 'dispatcher_areas' with pid 13
2019-03-05 21:42:32,847 INFO spawned: 'areas_rules' with pid 18
2019-03-05 21:42:32,872 INFO exited: fetch_diff (exit status 1; not expected)
2019-03-05 21:42:33,876 INFO success: overpass_dispatch entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-05 21:42:33,876 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-05 21:42:33,876 INFO success: fcgiwrap entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-05 21:42:33,876 INFO success: apply_diff entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-05 21:42:33,878 INFO spawned: 'fetch_diff' with pid 44
2019-03-05 21:42:33,878 INFO success: dispatcher_areas entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-05 21:42:33,878 INFO success: areas_rules entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-05 21:42:33,892 INFO exited: fetch_diff (exit status 1; not expected)
2019-03-05 21:42:35,896 INFO spawned: 'fetch_diff' with pid 53
2019-03-05 21:42:35,909 INFO exited: fetch_diff (exit status 1; not expected)
2019-03-05 21:42:38,914 INFO spawned: 'fetch_diff' with pid 67
2019-03-05 21:42:38,928 INFO exited: fetch_diff (exit status 1; not expected)
2019-03-05 21:42:38,928 INFO gave up: fetch_diff entered FATAL state, too many start retries too quickly

Installation went well but no data

Hi, I created an instance using this command :

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -v /big/docker/overpass_clone_db/:/db \
  -p 12346:80 \
  -i -t \
  --name overpass_world \
  wiktorn/overpass-api

Everything went well and I was asked to launch the container with docker start.

 database ready.
Overpass ready, you can start your container with docker start

I did it and tried to do a request using a Python, that is normally working on the regular overpass API.

response = overpy.Overpass(url="http://localhost:8082/api/interpreter").query(
    """
[out:json][timeout:900];
area["name"="Boston"]->.boundaryarea;
(
 node(area.boundaryarea)["amenity"="university"];
 way(area.boundaryarea)["amenity"="university"];>;
 relation(area.boundaryarea)["amenity"="university"];>>;
);
out meta;
"""
)

Unfortunately, I get no response. I don't have any error but it's like the database is empty although there is 169G inside the directory containing the database.

When I check the logs of the container, I see my requests, along with the minutely updates and other stuff. Here is an extract :

Reading XML file ... finished reading ways. After 0h19m46s: in "recurse", part 0, on line 26. Stack: 0 of 0 101297 of 0 0 of 0
After 0h20m1s: in "recurse", part 0, on line 26. Stack: 0 of 0 103524 of 0 0 of 0
After 0h20m16s: in "recurse", part 0, on line 26. Stack: 0 of 0 105693 of 0 0 of 0
After 0h20m31s: in "recurse", part 0, on line 26. Stack: 0 of 0 107515 of 0 0 of 0
After 0h20m46s: in "make-area", part 0, on line 29. Stack: 0 of 0 109766 of 0
Flushing to database ...After 0h21m1s: in "recurse", part 0, on line 26. Stack: 0 of 0 110915 of 0 0 of 0
After 0h21m16s: in "make-area", part 0, on line 29. Stack: 0 of 0 112117 of 0
After 0h21m31s: in "recurse", part 0, on line 26. Stack: 0 of 0 112888 of 0 0 of 0
.132.227.123.9 - - [29/Jan/2020:17:16:08 +0000] "POST /api/interpreter HTTP/1.1" 200 298 "-" "Python-urllib/3.6"
After 0h21m46s: in "make-area", part 0, on line 29. Stack: 0 of 0 114677 of 0
After 0h22m1s: in "recurse", part 0, on line 27. Stack: 0 of 0 116832 of 0 0 of 0
After 0h22m16s: in "recurse", part 0, on line 27. Stack: 0 of 0 118522 of 0 0 of 0
After 0h22m31s: in "recurse", part 0, on line 27. Stack: 0 of 0 120368 of 0 0 of 0
After 0h22m46s: in "recurse", part 0, on line 26. Stack: 0 of 0 122218 of 0 0 of 0
After 0h23m1s: in "recurse", part 0, on line 26. Stack: 0 of 0 124587 of 0 0 of 0
After 0h23m16s: in "recurse", part 0, on line 26. Stack: 0 of 0 126290 of 0 0 of 0
After 0h23m31s: in "recurse", part 0, on line 26. Stack: 0 of 0 128083 of 0 0 of 0

I'm I missing something ?

Best,
Matthieu.

It does not look like the update is working

Update finished with status code: 3 INFO: Using replication server at http://download.geofabrik.de/europe/germany/baden-wuerttemberg/tue/ DEBUG: Using given sequence ID 2593 DEBUG: Starting download at ID 2594 (max 100 MB) Empty version, skipping file

runtime error: open64: 2 No such file or directory /db/db/node_changelog.bin

I tried to run OverpassApi locally with the following command:

docker run \
    -e OVERPASS_META=yes \
    -e OVERPASS_MODE=init \
    -e OVERPASS_PLANET_URL=http://download.geofabrik.de/<foo>.pbf \
    -e OVERPASS_RULES_LOAD=10 \
    -e OVERPASS_COMPRESSION=gz \
    -e OVERPASS_UPDATE_SLEEP=3600 \
    -e OVERPASS_PLANET_PREPROCESS='mv /db/planet.osm.bz2 /db/planet.osm.pbf && osmium cat -o /db/planet.osm.bz2 /db/planet.osm.pbf && rm /db/planet.osm.pbf' \
    -v /big/docker/overpass_db/:/db \
    -p 12347:80 \
    -i -t \
    --name overpass_monaco wiktorn/overpass-api

as you see I did'nt find OVERPASS_DIFF_URL for the foo so I removed the OVERPASS_DIFF_URL and OVERPASS_COOKIE_JAR_CONTENTS env from docker run command, I just wanted to test overpass and I dont want diff updates right now, so it should not be important,
but after downloading data and completion of initialization, I started the container and tried a sample query but every time I get this result:

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.55.7 8b86ff77">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2019-09-30T14:43:02Z"/>

<remark> runtime error: Tried to use museum file but no museum files available on this instance. </remark>
<remark> runtime error: open64: 2 No such file or directory /db/db/node_changelog.bin File_Blocks::File_Blocks::1 </remark>

</osm>

now my question is, was the removal of env variable caused this error? if so is there any way to run overpass without diff updates?
Thanks.

Missing Backslash in README.md

  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/monaco-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/monaco/minute/ \
  -e OVERPASS_PLANET_SEQUENCE_ID=2946200 \
  -e OVERPASS_RULES_LOAD=10 \
  -v /big/docker/overpass_db/:/db  <-------- MISSING BACKSLASH HERE
  -p 12345:80 \
  -i -t \
  --name overpass_monaco wiktorn/overpass-api

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.