wiktorn / overpass-api Goto Github PK

View Code? Open in Web Editor NEW

130.0 6.0 47.0 217 KB

Overpass API docker image

License: MIT License

Shell 86.27% Python 13.73%

overpass-api's Issues

Inconsistent behavior using docker-compose

Issue:

When trying to access /api(http://localhost:12345) from the host machine, not a single page loads with the response:

localhost didn’t send any data.
ERR_EMPTY_RESPONSE

Steps to replicate:
Chrome: Version 87.0.4280.88 (Official Build) (64-bit)
OS: Windows, Docker for Windows 3.0.0 (50684), Engine: 20.10.0, WSL2 (20.04.1 LTS)
Overpassversion: tested both on 0.7.56 and latest tag

The following docker command works as described in the documentation works:

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/austria-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=https://download.openstreetmap.fr/replication/europe/austria/minute/ \
  -e OVERPASS_RULES_LOAD=10 \
  -v /home/neatnerd/backend/data/overpass/:/db \
  -p 12345:80 \
  -i -t \
  --name overpass_austria wiktorn/overpass-api

The following docker-compose file does not work:

version: "3"

services:
  overpass:
    image: wiktorn/overpass-api
    ports: 
      - "12345:8080"
    volumes:
      - ./data/overpass/:/db
    environment: 
      - OVERPASS_META=yes
      - OVERPASS_MODE=init
      - OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/austria-latest.osm.bz2
      - OVERPASS_DIFF_URL=https://download.openstreetmap.fr/replication/europe/austria/minute/
      - OVERPASS_RULES_LOAD=10

I have tried to switch off the firewall and anti-virus. The most strange behavior is that I believe the page pops up on startup and then goes down, but this might be browser cache.

Logs

Starting supervisord process

2020-12-25 08:49:30,870 CRIT Supervisor running as root (no user in config file)
2020-12-25 08:49:30,872 INFO supervisord started with pid 1
2020-12-25 08:49:31,875 INFO spawned: 'overpass_dispatch' with pid 10
2020-12-25 08:49:31,877 INFO spawned: 'nginx' with pid 11
2020-12-25 08:49:31,880 INFO spawned: 'fcgiwrap' with pid 12
2020-12-25 08:49:31,884 INFO spawned: 'update_overpass' with pid 13
2020-12-25 08:49:31,889 INFO spawned: 'dispatcher_areas' with pid 14
2020-12-25 08:49:31,892 INFO spawned: 'areas_rules' with pid 15
/app/bin /
/db/diffs/changes.osm exists. Trying to apply again.
XML parsing error at line 3, column 0: no element found
2020-12-25 08:49:32,929 INFO success: overpass_dispatch entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-12-25 08:49:32,929 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-12-25 08:49:32,929 INFO success: fcgiwrap entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-12-25 08:49:32,929 INFO success: update_overpass entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-12-25 08:49:32,929 INFO success: dispatcher_areas entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-12-25 08:49:32,929 INFO success: areas_rules entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.56.3 eb200aeb">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2020-12-23T23:31:53Z" areas="2020-12-23T23:31:53Z"/>
After 0h0m15s: in "recurse", part 0, on line 26. Stack: 0 of 0 518 of 0 0 of 0
After 0h0m30s: in "recurse", part 0, on line 27. Stack: 0 of 0 1449 of 0 0 of 0
After 0h0m45s: in "make-area", part 0, on line 29. Stack: 0 of 0 2394 of 0
After 0h1m0s: in "recurse", part 0, on line 27. Stack: 0 of 0 3255 of 0 0 of 0
/db/diffs/changes.osm exists. Trying to apply again.

Desired behavior

Docker-compose settings shall result in the same outcome as running individual docker commands

Detailed steps for using full attic data?

Has anyone successfully setup an Overpass API server with full (same as Overpass DE i.e. from 2012) attic data (for a certain region / country)? I noticed an earlier issue #5 touching this topic, with reference to instructions in Overpass wiki.

However after reading those instructions as well as some further references around the question (e.g. another wiki entry) I was not sure whether this is feasible and how much effort it would require in terms of learning all the details of the process, obtaining all data files etc.

Has someone done this and if so, could provide simple "for dummies" instructions? Or at least tell what kind of effort to expect to make it work?

(For my use case I'm actually interested mostly on historical "attic" data rather than applying new diffs to stay up-to-date...)

Unable to initiate docker

Hi,
Tried to initiate the docker like in the example.
I think the example referring to ./docker-entrypoint.sh? Didn't find any Docker file to run.
Also when did that, the script prompt:

Help would be very much appreciated!

very limited max connexion ? only 4 per seconds ?

i have up to 20 req/s to my overpass server using you docker image, and it quickly send HTTP 502 error with this message :

[error] 19#19: *717 connect() to unix:/nginx/fcgiwrap.socket failed (11: Resource temporarily unavailable) while connecting to upstream

I try to change a lot of nginx configuration, but none is working.

To test it, i have a Gatling script that send 8 request / seconds.
sample request :
[timeout:180][out:json];way(48.4655000,7.7280000,48.4656000,7.7281990)[highway];(._;>;);out;

Gatling result show that the serveur only reply 4 time / seconds. never more.

Area query results different from Overpass Turbo

Here's my query to collect all of the States within the US:

[out:json];
area['admin_level'='2']['name'='United States'];
(relation['admin_level'='4'](area););
out tags;

My Overpass server returns Baja California in the results for this query, even though this state is in Mexico. Overpass Turbo correctly excludes this from the results.

I'm using the wiktorn/overpass-api Docker image that I started it with:

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -v /home/ubuntu/overpass_clone_db/:/db \
  -p 80:80 \
  -i -t \
  --name overpass_world \
  wiktorn/overpass-api

Is there something that I can do to correct these results?
Are there certain places I can look for signs of trouble?

I've also posted this in StackOverflow, GIS StackExchange, and OpenStreetMap Help.

No such file or directory /db/db//osm3s_v0.7.55_areas Unix_Socket::7

Hello! First of all thank you for this work and for your support 👍

I have this issue with the container and that prevents areas queries to work. I read here and there and you said multiple times that areas are calculated as part of the execution o the container. In the initialization phase, I have seen multiple times messages about different parts of the "pipeline" being processed but, once it stopped and I restarted it, I see all "good" messages on the stdout/stderr except one nasty line:

runtime error: open64: 2 No such file or directory /db/db//osm3s_v0.7.55_areas Unix_Socket::7

Of course such file does not exist. Is there a way to trigger a make-area stage? Is there a way to see whether there's one running? How can I best troubleshoot why this file is not there and why I get this error? Thanks!

I tried the following command:

curl -g 'http://localhost:12345/api/interpreter?data=[out:json];area[name="Berlin"];out;'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
  <meta http-equiv="content-type" content="text/html; charset=utf-8" lang="en"/>
  <title>OSM3S Response</title>
</head>
<body>

<p>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</p>
<p><strong style="color:#FF0000">Error</strong>: runtime error: open64: 2 No such file or directory /db/db//osm3s_v0.7.55_areas Unix_Socket::7 </p>

</body>
</html>

I have bootstrapped the container with only Germany and the rules_loop.log is filled with "updates started/finished" messages.

Deploy to Kubernetes?

I'm trying to deploy this to (google) kubernetes but it's not working. Any ideas on what I need to change in this yaml?

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: overpass
spec:
  template:
    metadata:
      labels:
        name: overpass
    spec:
      containers:
        - name: overpass
          image: wiktorn/overpass-api
          volumeMounts:
          - mountPath: /db
            name: overpass-volume
          readinessProbe:
            httpGet:
              path: /interpreter
              port: 80
          ports:
            - name: overpass
              containerPort: 80
          env:
          - name: OVERPASS_META
            value: "no"
          - name: OVERPASS_MODE
            value: "init"
          - name: OVERPASS_PLANET_URL
            value: "http://download.geofabrik.de/australia-oceania/australia-latest.osm.bz2"
          - name: OVERPASS_DIFF_URL
            value: "http://download.openstreetmap.fr/replication/oceania/australia/minute/"
          - name: OVERPASS_PLANET_SEQUENCE_ID
            value: "3325745"
          - name: OVERPASS_RULES_LOAD
            value: "10"
      volumes:
      - name: overpass-volume
        emptyDir: {}

Trying to install this docker image on digital ocean with attached volume [question]

Thanks for creating this docker image, works great with smaller datasets. However we're now trying to set this up on digital ocean with an attached volume mounted on host machine /mnt/... However the install script doesnt seem to pick it up when I run it like below:

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/minute/ \
  -e OVERPASS_PLANET_SEQUENCE_ID=2946200 \
  -e OVERPASS_RULES_LOAD=10 \
  -v /big/docker/overpass_db/:/mnt/volume_nyc3_01/db \
  -p 12345:80 \
  -i -t \
  --name overpass_europe wiktorn/overpass-api

Any idea what we're doing wrong and how to modify it so the DB is stored on the mounted volume?

Thanks!

First install

Hi, please,
i just try to install an image with command in README

sudo docker run \
  -e OVERPASS_META=no \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -v /big/docker/overpass_clone_db/:/db \
  -p 8083:80 \
  -i -t \
  --name overpass_world \
  wiktorn/overpass-api

but i see only this:

2019-05-24 10:55:05,351 CRIT Supervisor running as root (no user in config file)
2019-05-24 10:55:05,355 INFO supervisord started with pid 1
2019-05-24 10:55:06,359 INFO spawned: 'overpass_dispatch' with pid 9
2019-05-24 10:55:06,361 INFO spawned: 'nginx' with pid 10
2019-05-24 10:55:06,364 INFO spawned: 'fcgiwrap' with pid 11
2019-05-24 10:55:06,367 INFO spawned: 'update_overpass' with pid 12
2019-05-24 10:55:06,370 INFO spawned: 'dispatcher_areas' with pid 15
2019-05-24 10:55:06,372 INFO spawned: 'areas_rules' with pid 16
2019-05-24 10:55:06,376 INFO exited: overpass_dispatch (exit status 0; not expected)
2019-05-24 10:55:07,561 INFO spawned: 'overpass_dispatch' with pid 43
2019-05-24 10:55:07,562 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-24 10:55:07,562 INFO success: fcgiwrap entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-24 10:55:07,562 INFO success: update_overpass entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-24 10:55:07,562 INFO success: dispatcher_areas entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-24 10:55:07,562 INFO success: areas_rules entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-24 10:55:07,574 INFO exited: overpass_dispatch (exit status 0; not expected)
2019-05-24 10:55:10,411 INFO spawned: 'overpass_dispatch' with pid 54
2019-05-24 10:55:10,424 INFO exited: overpass_dispatch (exit status 0; not expected)
2019-05-24 10:55:14,426 INFO spawned: 'overpass_dispatch' with pid 64
2019-05-24 10:55:14,439 INFO exited: overpass_dispatch (exit status 0; not expected)
2019-05-24 10:55:15,431 INFO gave up: overpass_dispatch entered FATAL state, too many start retries too quickly

Should i wait or what is wrong? /big/docker/ directory change only changes.log but no data downloaded.

In changes.log i is:


Traceback (most recent call last):
  File "/app/venv/bin/pyosmium-get-changes", line 243, in <module>
    exit(main(sys.argv[1:]))
  File "/app/venv/bin/pyosmium-get-changes", line 186, in main
    with open(options.seq_file, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/db/replicate_id'
/app/bin/update_database --compression-method= --map-compression-method= --flush-size= --db-dir=/db/db
cat: /db/diffs/changes.osm: No such file or directory
For --compression-method, please use "no", "gz", or "lz4" as value.
For --map-compression-method, please use "no", "gz", or "lz4" as value.
Usage: /app/bin/update_database [--db-dir=DIR] [--version=VER] [--meta|--keep-attic] [--flush_size=FLUSH_SIZE] [--compression-method=(no|gz|lz4)] [--map-compression-method=(no|gz|lz4)]

Missing Backslash in README.md

  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/monaco-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/monaco/minute/ \
  -e OVERPASS_PLANET_SEQUENCE_ID=2946200 \
  -e OVERPASS_RULES_LOAD=10 \
  -v /big/docker/overpass_db/:/db  <-------- MISSING BACKSLASH HERE
  -p 12345:80 \
  -i -t \
  --name overpass_monaco wiktorn/overpass-api

Allow resume of initalization mode

This line of the docker-entrypoint.sh file prohibits us from resuming an initial build that has failed/stopped, but the wget usage in download_clone.sh allows restarting.

Perhaps it's still safe if this line changes to:

if [[ ! -d /db/db/diffs ]] ; then

Welcome to nginx

Ok, this is probably a really stupid question but I'm pretty new to all this, 'how do I use this'?

I've been using https://overpass-turbo.eu/ but keep hitting rate limiting and query timeouts so decided to give wiktorn/Overpass-API a try. I initialized with the following command:

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -v ${HOME}/overpass:/db \
  -p 12346:80 \
  -i -t \
  --name overpass_world \
  wiktorn/overpass-api

I had to leave the office that evening so suspended my machine, leaving one of the downloads hanging.

When I returned in the morning, I restarted the container, and opened localhost:12346 - I was greeted by the "Welcome to Nginx" page.

The logs looked like this :

2018-06-22 09:28:28,329 CRIT Supervisor running as root (no user in config file)
2018-06-22 09:28:28,334 INFO supervisord started with pid 1                   
2018-06-22 09:28:29,339 INFO spawned: 'overpass_dispatch' with pid 9    
2018-06-22 09:28:29,341 INFO spawned: 'nginx' with pid 10               
2018-06-22 09:28:29,343 INFO spawned: 'fcgiwrap' with pid 11          
2018-06-22 09:28:29,345 INFO spawned: 'apply_diff' with pid 12
2018-06-22 09:28:29,347 INFO spawned: 'fetch_diff' with pid 15
2018-06-22 09:28:29,350 INFO spawned: 'dispatcher_areas' with pid 18           
2018-06-22 09:28:29,354 INFO spawned: 'areas_rules' with pid 19                 
2018-06-22 09:28:29,380 INFO exited: apply_diff (exit status 1; not expected)
2018-06-22 09:28:31,135 INFO success: overpass_dispatch entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-06-22 09:28:31,135 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-06-22 09:28:31,135 INFO success: fcgiwrap entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-06-22 09:28:31,139 INFO spawned: 'apply_diff' with pid 47
2018-06-22 09:28:31,140 INFO success: fetch_diff entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-06-22 09:28:31,140 INFO success: dispatcher_areas entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-06-22 09:28:31,141 INFO success: areas_rules entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-06-22 09:28:32,427 INFO success: apply_diff entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

I entered the container, executed the download_clone command, and allowed it to complete.

docker exec -ti overpass_world bash
/app/bin/download_clone.sh --db-dir=/db/db --source=http://dev.overpass-api.de/api_drolbr/ --meta=yes

I then stopped and started the container, but still just get the "Welcome to Nginx" home page.

I just went to http://overpass-turbo.eu (changing from https to http) - and changed the settings so it would send reqests to http://localhost:12346/api/

Tested it, and tada ! it worked !

So I suggest document how to use (although its obvious to anyone in the community ) and perhaps add my workaround for the failed initial download.

Failed to build docker

Trying to build a latest docker image locally with following docker-compose:

overpass:
    container_name: overpass
    build: 0.7.55.7/.
    cpu_shares: 2048
    mem_limit: 4096M
    environment:
        - OVERPASS_META=yes
        - OVERPASS_MODE=init
        - OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/monaco-latest.osm.bz2
        - OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/monaco/minute/
        - OVERPASS_RULES_LOAD=10
    ports:
        - 12345:80
    volumes:
        - ~/dump/db/:/db:rw

getting this:

/bin/bash ./libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -O2 -MT template_db/dispatcher_client.lo -MD -MP -MF $depbase.Tpo -c -o template_db/dispatcher_client.lo template_db/dispatcher_client.cc &&
mv -f $depbase.Tpo $depbase.Plo
libtool: compile: g++ -DHAVE_CONFIG_H -I. -O2 -MT template_db/dispatcher_client.lo -MD -MP -MF template_db/.deps/dispatcher_client.Tpo -c template_db/dispatcher_client.cc -fPIC -DPIC -o template_db/.libs/dispatcher_client.o
libtool: compile: g++ -DHAVE_CONFIG_H -I. -O2 -MT template_db/dispatcher_client.lo -MD -MP -MF template_db/.deps/dispatcher_client.Tpo -c template_db/dispatcher_client.cc -o template_db/dispatcher_client.o >/dev/null 2>&1
/bin/bash ./libtool --tag=CXX --mode=link g++ -O2 -o libexpatwrapper.la expat/expat_justparse_interface.lo -lexpat -lrt -lexpat
libtool: error: 'expat/expat_justparse_interface.lo' is not a valid libtool object
Makefile:1102: recipe for target 'libexpatwrapper.la' failed
make[1]: *** [libexpatwrapper.la] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory '/app/src'
Makefile:1873: recipe for target 'install-recursive' failed
make: *** [install-recursive] Error 1
ERROR: Service 'overpass' failed to build: The command '/bin/sh -c mkdir -p /app/src && cd /app/src && tar -x -z --strip-components 1 -f ../src.tar.gz && autoscan && aclocal && autoheader && libtoolize && automake --add-missing && autoconf && CXXFLAGS='-O2' CFLAGS='-O2' ./configure --prefix=/app --enable-lz4 && make -j $(grep -c ^processor /proc/cpuinfo) dist install clean && mkdir -p /db/diffs /app/etc && cp -r /app/src/rules /app/etc/rules && rm -rf /app/src /app/src.tar.gz' returned a non-zero code: 2

Log Nginx Errors

Sometimes I receive wird Nginx errors when accessing the Overpass instance. The error contains a hint to check the Nginx error log, but I don't think the container is writing logs? Maybe it would be possible to also write the Nginx logs to the disk?

Error when attempting to use attic data

Hi,
For some reason, I can't get get queries to work that use the date (attic) feature.

For wget -O - "http://192.168.1.147:12345/api/interpreter?data=[date:\"2017-10-28T19:20:00Z\"];node[\"addr:housenumber\"];out;"

I get the following result/error:

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.55 579b1eec">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2018-04-28T07:13:02Z"/>
<remark> runtime error: open64: 2 No such file or directory /db/db/node_tags_global_attic.bin File_Blocks::File_Blocks::1 </remark>
</osm>

By removing [date:\"2017-10-28T19:20:00Z\"]; from the query, OSM data is returned.

Could someone have a look at this issue?
Thank you

Steps to reproduce:

I followed the README example to start an overpass-api instance for Monaco with OVERPASS_META=attic.
I also tried OVERPASS_META=yes, but it made no difference.

docker run \
  -e OVERPASS_META=attic \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/monaco-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/monaco/minute/ \
  -e OVERPASS_PLANET_SEQUENCE_ID=2946200 \
  -e OVERPASS_RULES_LOAD=10 \
  -v /data/docker/overpass_db/:/db \
  -p 12345:80 \
  -i -t \
  --name overpass_monaco wiktorn/overpass-api

docker start overpass_monaco

wget -O - "http://192.168.1.147:12345/api/interpreter?data=[date:\"2017-10-28T19:20:00Z\"];node[\"addr:housenumber\"];out;"

wget -O - "http://192.168.1.147:12345/api/interpreter?data=node[\"addr:housenumber\"];out;"

docker build failed

see commit c9eb697#comments

502 response error - runtime error

Hi again,

Sorry for the constant issues.

I've run it with the following docker command (and subsequently started it):

docker run \ -e OVERPASS_META=yes \ -e OVERPASS_MODE=init \ -e OVERPASS_PLANET_URL=http://download.geofabrik.de/australia-oceania/new-zealand-latest.osm.bz2 \ -e OVERPASS_DIFF_URL=http://download.geofabrik.de/australia-oceania/new-zealand-updates/ \ -e OVERPASS_RULES_LOAD=10 \ -e OVERPASS_UPDATE_SLEEP=3600 \ -v E:\ecan\deployments\overpass\NZ\:/db \ -p 12345:80 \ -i -t \ --name overpass_nz wiktorn/overpass-api

Whenever I send a request to the overpass api I get a 502 response error with the following content:

<?xml version="1.0" encoding="UTF-8"?>\n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"\n "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">\n<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">\n<head>\n <meta http-equiv="content-type" content="text/html; charset=utf-8" lang="en"/>\n <title>OSM3S Response</title>\n</head>\n<body>\n\nThe data included in this document is from www.openstreetmap.org. The data is made available under ODbL.\nError: runtime error: The dispatcher (i.e. the database management system) is turned off. \n\n</body>\n</html>

It says the dispatcher is turned off...I am running the container...

Thanks again.

How to run on local osm file?

How to avoid using OVERPASS_PLANET_URL? I have generated an osm bz2 file of a specific region.

Problem after importing data

I tried to create an overpass instance out of tags wiktorn/overpass-api:latest and wiktorn/overpass-api:0.7.55 but after importing, the container seems to fail.

Here's what I did:

docker volume create overpass
docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/germany/nordrhein-westfalen/muenster-regbez-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -v overpass:/db \
  -p 12345:80 \
  -i -t \
  --name overpass_muenster wiktorn/overpass-api

docker start overpass_muenster

The logs of the overpass_muenster container are:

Reading XML file ... finished reading nodes. Flushing to database ...... done.                                            
Reading XML file ... elapsed way 320277976. Flushing to database ...... done.
Reading XML file ... finished reading ways. Flushing to database ...... done.
Reorganizing the database ... done.
Reading XML file ... finished reading relations. compute_geometry: Way 6654659 used in relation 4531 not found.
compute_geometry: Way 6650667 used in relation 4531 not found.
compute_geometry: Way 6651330 used in relation 4531 not found.
compute_geometry: Way 6651327 used in relation 4531 not found.
compute_geometry: Way 112448606 used in relation 4531 not found.

... many more of these ...

compute_geometry: Way 32731138 used in relation 9332852 not found.
compute_geometry: Way 30016603 used in relation 9332852 not found.
compute_geometry: Way 528727430 used in relation 9332852 not found.
compute_geometry: Way 37560917 used in relation 9332852 not found.
compute_geometry: Way 329748412 used in relation 9332852 not found.
Flushing to database ....... done.
Update complete.
Overpass ready, you can start your container with docker start
2019-03-05 21:42:31,815 CRIT Supervisor running as root (no user in config file)
2019-03-05 21:42:31,820 INFO supervisord started with pid 1
2019-03-05 21:42:32,824 INFO spawned: 'overpass_dispatch' with pid 8
2019-03-05 21:42:32,828 INFO spawned: 'nginx' with pid 9
2019-03-05 21:42:32,835 INFO spawned: 'fcgiwrap' with pid 10
2019-03-05 21:42:32,837 INFO spawned: 'apply_diff' with pid 11
2019-03-05 21:42:32,839 INFO spawned: 'fetch_diff' with pid 12
2019-03-05 21:42:32,841 INFO spawned: 'dispatcher_areas' with pid 13
2019-03-05 21:42:32,847 INFO spawned: 'areas_rules' with pid 18
2019-03-05 21:42:32,872 INFO exited: fetch_diff (exit status 1; not expected)
2019-03-05 21:42:33,876 INFO success: overpass_dispatch entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-05 21:42:33,876 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-05 21:42:33,876 INFO success: fcgiwrap entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-05 21:42:33,876 INFO success: apply_diff entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-05 21:42:33,878 INFO spawned: 'fetch_diff' with pid 44
2019-03-05 21:42:33,878 INFO success: dispatcher_areas entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-05 21:42:33,878 INFO success: areas_rules entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-05 21:42:33,892 INFO exited: fetch_diff (exit status 1; not expected)
2019-03-05 21:42:35,896 INFO spawned: 'fetch_diff' with pid 53
2019-03-05 21:42:35,909 INFO exited: fetch_diff (exit status 1; not expected)
2019-03-05 21:42:38,914 INFO spawned: 'fetch_diff' with pid 67
2019-03-05 21:42:38,928 INFO exited: fetch_diff (exit status 1; not expected)
2019-03-05 21:42:38,928 INFO gave up: fetch_diff entered FATAL state, too many start retries too quickly

Cloning uses older dataset of two available

I am trying to install the latest image using this command```

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -v /Volumes/Test/Docker/overpass_clone_db/:/db \
  -p 12346:80 \
  -i -t \
  --name overpass_world \
  wiktorn/overpass-api

There are two images on https://dev.overpass-api.de/clone/ at the moment:


  | Name | Last modified | Size | Description
  | 2020-06-11/ | 2020-06-11 09:01 | - |  
  | 2020-06-12/ | 2020-06-12 05:44 | - |

When I start the image installation, it starts to download the old image (2020-06-11) though instead of the newer one. I need the newer one as download takes more than 24 hours on my internet connection and the old image will disappear within that time.

Rationale behind having to start the container twice?

I see that after the database is initialized, the script exits and tells you to run 'docker start', such that on the second run it will start nginx.

Is there a reason for that, rather than just starting nginx after initialisation?

Area search doesn't seem to work

After a 24 hours parsing of the Europe dataset from Geofabrik I have the overpass api succesfully running inside the docker container.
Simple overpass queries seem to work just fine but for some reason area limiting doesn't seem to work, no error, just no results. Below query works fine on the public overpass-turbo server but not on my own instance.

{{geocodeArea:Germany}}->.country;
rel(area.country)["boundary"="administrative"]["name"~"Main"][admin_level=10];
out geom;

area["ISO3166-1"="DE"][admin_level=2];
rel["boundary"="administrative"]["name"~"Mainz"][admin_level=6](area);
out geom;

When I remove the area restriction it works but it returns all matches inside Europe.

Any idea?

Failed to process planet file

Hi,

I'm running into a problem running the docker image on a windows machine. I'm trying to run it for New Zealand using the geofabric site:

Gives me the error after initially processing the planet file:

No database directory. Initializing % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 373M 100 373M 0 0 7075k 0 0:00:54 0:00:54 --:--:-- 6675k [======================================================================] 100% Reading XML file ... elapsed node 2945583286. /app/bin/init_osm3s.sh: line 44: 16 Broken pipe bunzip2 < $PLANET_FILE 17 Killed | $EXEC_DIR/bin/update_database --db-dir=$DB_DIR/ $META $COMPRESSION Failed to process planet file

My guess is because it's a windows mount where the database sits rather than Linux? The sh script doesn't account for this?

Thanks!

What is the unit of OVERPASS_SPACE ? / Add unit for OVERPASS_SPACE

I am missing the documentation for OVERPASS_SPACE. What is the unit of this value?

Is it possible to clip diffs according to a polygon?

Hello,

We are interested only for a small region of central europe. We can extract it from planet file and use it for OVERPASS_PLANET_URL. Is it possible to apply our clipping polygon from GeoJSON file to minutely diffs?

Running the overpass-api image in a serverless environment

Hello,

I have been trying to see if I can run the image on Google Cloud Run - fully managed.

It seems that docker-entrypoint.sh does the heavy lifting of setting up overpass environment during the first run.

I was curious if this overpass-db setup process could be done as part of docker image building process, such that overpass-db is ready to be used with the image, and the entry point script will only start the nginx server.

This is because Google Cloud Run - fully managed does not provide a mount, and there is no concept of 'first run', 'non-first run' as everything is ephemeral except the image itself.

e.g. If I use the current image, it will attempt to initialize overpass-db by downloading planet file per every request.

Do you have any guidance on how to do that? (or if this cannot be done for some reason?)

Option for (currently static) nginx timeouts?

First of all @wiktorn , thank you for this Docker image! Even for someone who is just starting to learn Docker it was very easy to setup an overpass server with the help of this image.

While I was running some tests today querying my docker overpass instance through wget, I noticed that the connection to the server was reset roughly every 16 minutes. As my query needs to run longer (this is the main reason for my private overpass instance), wget then started a second query which resulted in the same two queries being executed on the server at the same time with an offset of 16 mins.

The culprit in this case is the fastcgi_read_timeout 1000s; directive in nginx.conf.

So my question/feature request is to pass that value as an argument to docker run if possible. Alternatively one could think about a higher default value (like 24 hours or so).

Thanks and best regards!

[Edit] Changed keepalive_timeout to fastcgi_read_timeout.

Update repeats in loop without sleep

After initialization using image version 0.7.56.8 (also tried 0.7.55.9) updating process repeats in loop without sleep. Initialization configuration:

docker run \
  -e OVERPASS_META=no \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/monaco-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/monaco/minute/ \
  -e OVERPASS_RULES_LOAD=10 \
  -e OVERPASS_UPDATE_SLEEP=300 \
  -v /mnt/OSM_data/overpass_monaco/:/db \
  -p 12345:80 \
  -i -t \
  --name overpass_monaco wiktorn/overpass-api:0.7.56.8

Also tried with OVERPASS_META=yes and without OVERPASS_RULES_LOAD or OVERPASS_UPDATE_SLEEP variables.
Logs output looks like:

log

There are still some updates remaining
2020-12-28 08:37:57 INFO: Using replication server at http://download.openstreetmap.fr/replication/europe/monaco/minute/
2020-12-28 08:37:57 DEBUG: Using given sequence ID 4346024
2020-12-28 08:37:57 DEBUG: Starting download at ID 4346025 (max 100 MB)
2020-12-28 08:37:57 DEBUG: Downloaded change 4346025. (102400 kB available in download buffer)
2020-12-28 08:37:57 DEBUG: Downloaded change 4346026. (102400 kB available in download buffer)
2020-12-28 08:37:57 DEBUG: Downloaded change 4346027. (102400 kB available in download buffer)
Empty version, skipping file
<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="libosmium/2.15.6">
</osm>
There are still some updates remaining
2020-12-28 08:37:57 INFO: Using replication server at http://download.openstreetmap.fr/replication/europe/monaco/minute/
2020-12-28 08:37:57 DEBUG: Using given sequence ID 4346024
2020-12-28 08:37:57 DEBUG: Starting download at ID 4346025 (max 100 MB)
2020-12-28 08:37:57 DEBUG: Downloaded change 4346025. (102400 kB available in download buffer)
2020-12-28 08:37:57 DEBUG: Downloaded change 4346026. (102400 kB available in download buffer)
2020-12-28 08:37:57 DEBUG: Downloaded change 4346027. (102400 kB available in download buffer)
Empty version, skipping file
<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="libosmium/2.15.6">
</osm>
There are still some updates remaining
2020-12-28 08:37:57 INFO: Using replication server at http://download.openstreetmap.fr/replication/europe/monaco/minute/
2020-12-28 08:37:57 DEBUG: Using given sequence ID 4346024
2020-12-28 08:37:57 DEBUG: Starting download at ID 4346025 (max 100 MB)
2020-12-28 08:37:57 DEBUG: Downloaded change 4346025. (102400 kB available in download buffer)
2020-12-28 08:37:58 DEBUG: Downloaded change 4346026. (102400 kB available in download buffer)
2020-12-28 08:37:58 DEBUG: Downloaded change 4346027. (102400 kB available in download buffer)
Empty version, skipping file
<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="libosmium/2.15.6">
</osm>
There are still some updates remaining

Though version latest which I pulled April 2020 seems to be working fine (it logs Overpass API 0.7.56.1003 though image wiktorn/overpass-api:0.7.56.1003 still loops without sleep). Started container using latest version from April 2020 sleeps after update showing appropriate message:

log

There are still some updates remaining
INFO: Using replication server at http://download.openstreetmap.fr/replication/europe/monaco/minute/
DEBUG: Using given sequence ID 4348922
DEBUG: Starting download at ID 4348923 (max 100 MB)
Empty version, skipping file
<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="libosmium/2.15.4">
</osm>
Update finished with status code: 3
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.56.1003 429967a9">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2020-12-30T01:00:55Z" areas="2020-12-30T01:00:55Z"/>


</osm>
It took 2 to run the loop. Desired load is: 1%. Sleeping: 198
<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.56.1003 429967a9">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2020-12-30T01:00:55Z" areas="2020-12-30T01:00:55Z"/>


</osm>
It took 2 to run the loop. Desired load is: 1%. Sleeping: 198
INFO: Using replication server at http://download.openstreetmap.fr/replication/europe/monaco/minute/
DEBUG: Using given sequence ID 4348922
DEBUG: Starting download at ID 4348923 (max 100 MB)

Example env vars incorrect?

I'm trying to run the example using the below, slightly modified code

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/poland-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/poland/minute/ \
  -e OVERPASS_PLANET_SEQUENCE_ID=2346486 \
  -p 12345:80 \
  -i -t \
  --name overpass_poland wiktorn/overpass-api:0.7.55

But it throws an error with the supervisord.conf

Error: Format string 'nice /app/bin/rules_loop.sh /db/db %(ENV_OVERPASS_RULES_LOAD)s' for 'program:areas_rules.command' contains names ('ENV_OVERPASS_RULES_LOAD') which cannot be expanded. Available names: ENV_HOME, ENV_HOSTNAME, ENV_NGINX_VERSION, ENV_NJS_VERSION, ENV_OVERPASS_DIFF_URL, ENV_OVERPASS_META, ENV_OVERPASS_MODE, ENV_OVERPASS_PLANET_SEQUENCE_ID, ENV_OVERPASS_PLANET_URL, ENV_PATH, ENV_PWD, ENV_SHLVL, ENV_TERM, group_name, here, host_node_name, process_num, program_name in section 'program:areas_rules' (file: '/etc/supervisor/conf.d/supervisord.conf')
For help, use /usr/bin/supervisord -h

Have the env var names changed?

Setting OVERPASS_META to "no" results in startup error.

This might be a regression from be98c86.

Setting the environment variable OVERPASS_META to no results in the following error message when trying to boot the dispatcher:

File /db/db/nodes_meta.bin present. Please use parameter --meta
File /db/db/ways_meta.bin present. Please use parameter --meta
File /db/db/relations_meta.bin present. Please use parameter --meta
File /db/db/user_data.bin present. Please use parameter --meta
File /db/db/user_indices.bin present. Please use parameter --meta

I believe this issue is here: https://github.com/wiktorn/Overpass-API/blob/master/docker-entrypoint.sh#L45

According to the help message, and this comment drolbr/Overpass-API#446 (comment) we should either omit or include the --meta flag.

Usage:  /app/bin/init_osm3s.sh  Planet_File  Database_Dir  Executable_Dir  [--meta]
        where
    Planet_File is the filename and path of the compressed planet file, including .bz2,
    Database_Dir is the directory the database should go into, and
    Executable_Dir is the directory that contains the executable update_database.
    Add --meta in the end if you want to use meta data.

Sometime update got stuck

Hi,

Sometime, the update process got stuck with this message (repeated durings hours ....) :

INFO: Using replication server at http://download.openstreetmap.fr/replication/europe/france/minute/
DEBUG: Using given sequence ID 4039576
DEBUG: Starting download at ID 4039577 (max 100 MB)
DEBUG: Loading state info None failed with: <urlopen error [Errno 99] Cannot assign requested address>
Empty version, skipping file
<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="libosmium/2.15.4">
</osm>
Update finished with status code: 3

it seems that pyosmium-get-changes got an error at certain point ?
but why ?

docker start bug

hello,
version: latest
desc: when i use docker start the overpass server , tips :

Reading XML file .../app/bin/init_osm3s.sh: line 44: 10 Broken pipe bunzip2 < $PLANET_FILE
11 Killed | $EXEC_DIR/bin/update_database --db-dir=$DB_DIR/ $META $COMPRESSION
how to fix it;
thanks!

How to use the predownloaded osm.bz file when initializing docker?

It is not really an issue, I would just like to know on how can we use local file as source for OSM_PLANET_URL parameter. I have read from the update that file:/// scheme is now supported in OSM_PLANET_URL parameter. I tried several ways but have encountered several errors like:

1.) OVERPASS_PLANET_URL=file://C:/file/path/to/sample.osm.bz2
error: curl: (3) Valid host name with slash missing in URL

2.) OVERPASS_PLANET_URL=file:///C:/file/path/to/sample.osm.bz2
error: curl: (37) Couldn't open file /C:/file/path/to/sample.osm.bz2

3.) OVERPASS_PLANET_URL='file:///C:/file/path/to/sample.osm.bz2'
error: curl: (1) Protocol "'file" not supported or disabled in libcurl

I hope you can provide some assistance. Thank you

Following the build instructions for the clone results in a 403 error on access

After following the middle instruction to clone a new world using the VM everything appears to work, but on accessing the port we get a 403 error from the Nginx running inside the docker instance. It looks like the files have downloaded correctly and docker does seem to have permissions to access the folder that it has downloaded the database files to.

If this happened to someone else, am I missing something obvious?

Minutely updater failing and not restarting with container

The minutely downloader seems to be crashing and it doesn't seem like it restarts when the docker instance is restarted.

I've noticed the minutely updater can be unstable in our previous (non-docker) instance so I know that it can be prone to crashing; I've also seen it restart from a previous point inside a docker instance so I'm confident it's configured correctly normally.

Are there any files I should grab off the container that might help in figuring this out? We'll keep it around for debugging purposes to triangulate this issue.

Permission denied

Hi,
First of all thank you for the container!
I have a problem with the container and can't find the cause. I start the container with the following command:
docker run
-e OVERPASS_META=yes
-e OVERPASS_MODE=init
-e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/germany/baden-wuerttemberg/stuttgart-regbez-latest.osm.bz2
-e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/germany/minute/
-e OVERPASS_RULES_LOAD=10
-v overpass:/db/
-p 12345:80
-i -t
--name overpass_bw wiktorn/overpass-api

The data is downloaded and then the container is finished. After the restart I get a permission error in the log:

It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
/app/bin/rules_loop.sh: line 42: ///db/db/rules_loop.log: Permission denied
/app/bin/rules_loop.sh: line 43: ///db/db/rules/areas.osm3s: No such file or directory
/app/bin/rules_loop.sh: line 44: ///db/db/rules_loop.log: Permission denied
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
/app/bin/rules_loop.sh: line 42: ///db/db/rules_loop.log: Permission denied
/app/bin/rules_loop.sh: line 43: ///db/db/rules/areas.osm3s: No such file or directory
/app/bin/rules_loop.sh: line 44: ///db/db/rules_loop.log: Permission denied
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
/app/bin/rules_loop.sh: line 42: ///db/db/rules_loop.log: Permission denied
/app/bin/rules_loop.sh: line 43: ///db/db/rules/areas.osm3s: No such file or directory
/app/bin/rules_loop.sh: line 44: ///db/db/rules_loop.log: Permission denied
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3

Response:
The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.
Error: runtime error: The dispatcher (i.e. the database management system) is turned off.

Is there any indication of what exactly the problem is?

Dummy metadata in Geofabrik public extract downloads.

The example "Overpass instance covering part of the world" uses a geofabrik file where all users have been mapped to a dummy user (user name = "", uid = 0) because of some GDPR requirements. Obviously, this is not ideal. To make this work as before, the extract needs to be downloaded from https://osm-internal.download.geofabrik.de/index.html --> "Login with your OpenStreetMap account "

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/monaco-latest.osm.bz2 \

overpass.errors.ServerRuntimeError: runtime error: Query ran out of memory

Hey I am getting the following error when quering the overpass API:

overpass.errors.ServerRuntimeError: runtime error: Query ran out of memory in "query" at line 1. It would need at least 517 MB of RAM to continue.

Since the server has 16GB of Ram and only 20% is in use - is there any way to give Overpass more memory ressources?

Greetings

Runtime error: The dispatcher (i.e. the database management system) is turned off. Kubernetes

I was able to run the container for australia region and the docker container restarted after downloading entire db and the pod showed 1/1 on kubernetes . However when I check the logs and query for the api . I get strange error as -

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
  <meta http-equiv="content-type" content="text/html; charset=utf-8" lang="en"/>
  <title>OSM3S Response</title>
</head>
<body>

<p>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</p>
<p><strong style="color:#FF0000">Error</strong>: runtime error: The dispatcher (i.e. the database management system) is turned off. </p>

</body>
</html>

Traceback (most recent call last):
  File "data_task__by_nikhil.py", line 140, in <module>
    data_frame = extract_raw_data_from_OSM(query)
  File "data_task__by_nikhil.py", line 118, in extract_raw_data_from_OSM
    json_data = response.json()
  File "/usr/lib/python3.8/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Below are the logs from the pod

It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
Update finished with status code: 3
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
10.240.0.115 - - [14/Nov/2020:13:22:49 +0000] "GET /api/status HTTP/1.1" 200 156 "-" "kube-probe/1.19"
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
10.240.0.115 - - [14/Nov/2020:13:22:59 +0000] "GET /api/status HTTP/1.1" 200 156 "-" "kube-probe/1.19"
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 1 to run the loop. Desired load is: 10%. Sleeping: 9
10.240.0.115 - - [14/Nov/2020:13:23:09 +0000] "GET /api/status HTTP/1.1" 200 156 "-" "kube-probe/1.19"
10.240.0.115 - - [14/Nov/2020:13:23:12 +0000] "GET /api/interpreter?data=%5Bout%3Ajson%5D%5Btimeout%3A50%5D%3B%28way%5B%22highway%22%5D%28around%3A50%2C-34.06221%2C150.82524%29%5Bmaxspeed%5D%3B%29%3Bout+body%3B HTTP/1.1" 504 635 "-" "python-requests/2.24.0"
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
10.240.0.115 - - [14/Nov/2020:13:23:19 +0000] "GET /api/status HTTP/1.1" 200 155 "-" "kube-probe/1.19"
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3
runtime error: The dispatcher (i.e. the database management system) is turned off.
It took 0 to run the loop. Desired load is: 10%. Sleeping: 3

Can you please guide how and why is this happening and how to resolve this ?

No (or empty) response form server - Request returns Error Code 403

Hey there,

if been trying to set up the overpass API for Germany. Therefore I used the provided script and modified it:
docker run \ -e OVERPASS_META=yes \ -e OVERPASS_MODE=init \ -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/germany-latest.osm.bz2 \ -e OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/germany/minute/ \ -e OVERPASS_RULES_LOAD=10 \ -v /big/docker/overpass_db/:/db \ -p 12345:80 \ -i -t \ --name overpass_germany wiktorn/overpass-api

it took a while but now everything is set up. I did the "docker start overpass_germany" comand and it booted. When i am now querying the server with "http;//<server_ip>:12345/api/ I do get error code 404.

Any ideas how I can fix it?

http.client.IncompleteRead

The get request in python is not working. I am hosting on a gcloud computing instance. With Postman and Curl the request is working, but not with python and the request lib.

Docker configuration:



docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -e OVERPASS_RATE_LIMIT=1 \
  -e OVERPASS_SPACE=8589934592 \
  -v /big/docker/overpass_clone_db/:/db \
  -p 12346:80 \
  -i -t \
  --name overpass_world \
  wiktorn/overpass-api

docker start overpass_world

Working:
curl --location -g --request GET 'http://35.242.198.27:12346/api/interpreter?data=[timeout:3600][maxsize:1073741824][out:json];node(around:5000,51.408159,7.2989356,51.4030306,7.3490601,51.4364065,7.3528927,51.4466813,7.3238974,51.408159,7.2989356)[name];out;%0A'

Not working:


url = "http://35.242.198.27:12346/api/interpreter?data=[timeout:3600][maxsize:1073741824][out:json];node(around:5000,51.408159,7.2989356,51.4030306,7.3490601,51.4364065,7.3528927,51.4466813,7.3238974,51.408159,7.2989356)[name];out;"

response = requests.request("GET", url)
#
print(response.text)`

Produces the following error:

Traceback (most recent call last):
  File "/home/julian/develop/dielage/backend/real-estate-integrator/venv/lib/python3.8/site-packages/urllib3/response.py", line 436, in _error_catcher
    yield
  File "/home/julian/develop/dielage/backend/real-estate-integrator/venv/lib/python3.8/site-packages/urllib3/response.py", line 766, in read_chunked
    chunk = self._handle_chunk(amt)
  File "/home/julian/develop/dielage/backend/real-estate-integrator/venv/lib/python3.8/site-packages/urllib3/response.py", line 710, in _handle_chunk
    value = self._fp._safe_read(amt)
  File "/usr/lib/python3.8/http/client.py", line 614, in _safe_read
    raise IncompleteRead(data, amt-len(data))
http.client.IncompleteRead: IncompleteRead(5461 bytes read, 4779 more expected)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/julian/develop/dielage/backend/real-estate-integrator/venv/lib/python3.8/site-packages/requests/models.py", line 751, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/home/julian/develop/dielage/backend/real-estate-integrator/venv/lib/python3.8/site-packages/urllib3/response.py", line 571, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/home/julian/develop/dielage/backend/real-estate-integrator/venv/lib/python3.8/site-packages/urllib3/response.py", line 792, in read_chunked
    self._original_response.close()
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/julian/develop/dielage/backend/real-estate-integrator/venv/lib/python3.8/site-packages/urllib3/response.py", line 454, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(5461 bytes read, 4779 more expected)', IncompleteRead(5461 bytes read, 4779 more expected))

During handling of the above exception, another exception occurred:

Memory issues when initializing container

Hi,

I tried to initialize a container using a tarball of whole Europe (~32GB) on my machine having 8GB of memory and an SSD disk.

I expexted it would take a while but not so much... it's running since yesterday. After downloading the snapshot, my linux box started swapping and all I see from the output of the container is a buch of lines like:

Reading XML file.... elapsed way 267461502. Flushing the database...... done. with the integer increasing
and Reorganizing the database.... done.

Do you recommend me killing the process and trying something else? Or should I wait? I'm currently on vacation so i dont have any problem letting the machine run for another day.

Is there a way to predict more or less how much time is going to require until the end?

Is there a way to tweak the init run to avoid running into swapping issues?

Any suggestion?

Thanks in advance

404 Not Found errors for retrieving Overpass data

I am running the clone example from the README. The docker image starts successfully, but after downloading data, I have many 404 Not Found errors for retrieving from the Overpass server.

kmehta@kmehta-desktop:~$ sudo docker run   -e OVERPASS_META=yes   -e OVERPASS_MODE=clone   -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/   -v /big/docker/overpass_clone_db/:/db   -p 12346:80   -i -t   --name overpass_world   wiktorn/overpass-api
No database directory. Initializing
--2021-01-19 18:46:58--  http://dev.overpass-api.de/api_drolbr//trigger_clone
Resolving dev.overpass-api.de (dev.overpass-api.de)... 95.217.37.171, 2a01:4f9:2b:2fd4::2
Connecting to dev.overpass-api.de (dev.overpass-api.de)|95.217.37.171|:80... connected.
HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable

    The file is already fully retrieved; nothing to do.

--2021-01-19 18:47:01--  https://dev.overpass-api.de/clone//2021-01-15/replicate_id
Resolving dev.overpass-api.de (dev.overpass-api.de)... 95.217.37.171, 2a01:4f9:2b:2fd4::2
Connecting to dev.overpass-api.de (dev.overpass-api.de)|95.217.37.171|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-01-19 18:47:02 ERROR 404: Not Found.


Fetching nodes.bin
--2021-01-19 18:47:02--  https://dev.overpass-api.de/clone//2021-01-15/nodes.bin
Resolving dev.overpass-api.de (dev.overpass-api.de)... 95.217.37.171, 2a01:4f9:2b:2fd4::2
Connecting to dev.overpass-api.de (dev.overpass-api.de)|95.217.37.171|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-01-19 18:47:02 ERROR 404: Not Found.

overpass log rotate setting inside docker image ?

overpass spawn a lot of logs in changes.log and transactions.log. (rules_loop.log also but it's a smaller size).

it is possible to use logrotate outside of the docker image to limit the size of thoses files. BUT it would be nice if the log rotate things is package inside the docker image .

do you think it's a good idea, or logrotate outside is the only good way to do it ?

runtime error: open64: 2 No such file or directory /db/db/node_changelog.bin

I tried to run OverpassApi locally with the following command:

docker run \
    -e OVERPASS_META=yes \
    -e OVERPASS_MODE=init \
    -e OVERPASS_PLANET_URL=http://download.geofabrik.de/<foo>.pbf \
    -e OVERPASS_RULES_LOAD=10 \
    -e OVERPASS_COMPRESSION=gz \
    -e OVERPASS_UPDATE_SLEEP=3600 \
    -e OVERPASS_PLANET_PREPROCESS='mv /db/planet.osm.bz2 /db/planet.osm.pbf && osmium cat -o /db/planet.osm.bz2 /db/planet.osm.pbf && rm /db/planet.osm.pbf' \
    -v /big/docker/overpass_db/:/db \
    -p 12347:80 \
    -i -t \
    --name overpass_monaco wiktorn/overpass-api

as you see I did'nt find OVERPASS_DIFF_URL for the foo so I removed the OVERPASS_DIFF_URL and OVERPASS_COOKIE_JAR_CONTENTS env from docker run command, I just wanted to test overpass and I dont want diff updates right now, so it should not be important,
but after downloading data and completion of initialization, I started the container and tried a sample query but every time I get this result:

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API 0.7.55.7 8b86ff77">
<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>
<meta osm_base="2019-09-30T14:43:02Z"/>

<remark> runtime error: Tried to use museum file but no museum files available on this instance. </remark>
<remark> runtime error: open64: 2 No such file or directory /db/db/node_changelog.bin File_Blocks::File_Blocks::1 </remark>

</osm>

now my question is, was the removal of env variable caused this error? if so is there any way to run overpass without diff updates?
Thanks.

Installation went well but no data

Hi, I created an instance using this command :

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -v /big/docker/overpass_clone_db/:/db \
  -p 12346:80 \
  -i -t \
  --name overpass_world \
  wiktorn/overpass-api

Everything went well and I was asked to launch the container with docker start.

 database ready.
Overpass ready, you can start your container with docker start

I did it and tried to do a request using a Python, that is normally working on the regular overpass API.

response = overpy.Overpass(url="http://localhost:8082/api/interpreter").query(
    """
[out:json][timeout:900];
area["name"="Boston"]->.boundaryarea;
(
 node(area.boundaryarea)["amenity"="university"];
 way(area.boundaryarea)["amenity"="university"];>;
 relation(area.boundaryarea)["amenity"="university"];>>;
);
out meta;
"""
)

Unfortunately, I get no response. I don't have any error but it's like the database is empty although there is 169G inside the directory containing the database.

When I check the logs of the container, I see my requests, along with the minutely updates and other stuff. Here is an extract :

Reading XML file ... finished reading ways. After 0h19m46s: in "recurse", part 0, on line 26. Stack: 0 of 0 101297 of 0 0 of 0
After 0h20m1s: in "recurse", part 0, on line 26. Stack: 0 of 0 103524 of 0 0 of 0
After 0h20m16s: in "recurse", part 0, on line 26. Stack: 0 of 0 105693 of 0 0 of 0
After 0h20m31s: in "recurse", part 0, on line 26. Stack: 0 of 0 107515 of 0 0 of 0
After 0h20m46s: in "make-area", part 0, on line 29. Stack: 0 of 0 109766 of 0
Flushing to database ...After 0h21m1s: in "recurse", part 0, on line 26. Stack: 0 of 0 110915 of 0 0 of 0
After 0h21m16s: in "make-area", part 0, on line 29. Stack: 0 of 0 112117 of 0
After 0h21m31s: in "recurse", part 0, on line 26. Stack: 0 of 0 112888 of 0 0 of 0
.132.227.123.9 - - [29/Jan/2020:17:16:08 +0000] "POST /api/interpreter HTTP/1.1" 200 298 "-" "Python-urllib/3.6"
After 0h21m46s: in "make-area", part 0, on line 29. Stack: 0 of 0 114677 of 0
After 0h22m1s: in "recurse", part 0, on line 27. Stack: 0 of 0 116832 of 0 0 of 0
After 0h22m16s: in "recurse", part 0, on line 27. Stack: 0 of 0 118522 of 0 0 of 0
After 0h22m31s: in "recurse", part 0, on line 27. Stack: 0 of 0 120368 of 0 0 of 0
After 0h22m46s: in "recurse", part 0, on line 26. Stack: 0 of 0 122218 of 0 0 of 0
After 0h23m1s: in "recurse", part 0, on line 26. Stack: 0 of 0 124587 of 0 0 of 0
After 0h23m16s: in "recurse", part 0, on line 26. Stack: 0 of 0 126290 of 0 0 of 0
After 0h23m31s: in "recurse", part 0, on line 26. Stack: 0 of 0 128083 of 0 0 of 0

I'm I missing something ?

Best,
Matthieu.

Dispatcher Client timeout

I successfully created an overpass instance with this Docker image about a month ago. When I was testing different parameters/configurations the past weeks, I deleted the working container and volume and created new containers. But since a few weeks they all fail to work.

The creation of the overpass instance seems to go through fine, nginx is responding to queries, but all queries time out and nginx returns a 504. Creating the overpass_monaco example does work. Using this or similar commands cloning the planet reproduce the timeout issue:

docker run \
  -e OVERPASS_META=yes \
  -e OVERPASS_MODE=clone \
  -e OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/ \
  -e OVERPASS_MAX_TIMEOUT=14400s \
  -e OVERPASS_SPACE=16777216 \
  -p 12345:80 \
  -d \
  --mount source=overpass,target=/db \
  --restart unless-stopped \
  --name overpass \
  wiktorn/overpass-api

The response from nginx is always

<p>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</p>
<p><strong style="color:#FF0000">Error</strong>: runtime error: open64: 0 Success /osm3s_v0.7.55_osm_base Dispatcher_Client::request_read_and_idx::timeout. The server is probably too busy to handle your request. </p>

"Too busy" cannot be the real reason as I am the only user running simple queries to test.

http://myhost.com:12345/api/status returns

Connected as: 1559890362
Current time: 2020-07-16T15:51:08Z
Rate limit: 0
Currently running queries (pid, space limit, time limit, start time):

so all good here.

changes.log looks OK. Updates are being made.
transactions.log looks a bit suspicious:

2020-07-16 15:17:20 [16] Dispatcher version 0.7.56.3 eb200aebdae27bd8635c5829a2bdfcdb06d0ab5c just started.
2020-07-16 15:17:20 [15] Dispatcher version 0.7.56.3 eb200aebdae27bd8635c5829a2bdfcdb06d0ab5c just started.
2020-07-16 15:17:20 [33] request_read_and_idx() start
2020-07-16 15:17:21 [16] hangup of process 27.
2020-07-16 15:17:21 [16] waited idle for 8 cycles.
2020-07-16 15:17:21 [16] waited idle for 8 cycles.
[...]
2020-07-16 15:17:56 [16] waited idle for 8 cycles.
2020-07-16 15:17:57 [33] Dispatcher_Client::request_read_and_idx::timeout /osm3s_v0.7.55_osm_base 0 Success 
2020-07-16 15:17:57 [16] waited idle for 8 cycles.
2020-07-16 15:17:57 [16] hangup of process 33.
2020-07-16 15:18:14 [818] write_start() start version='2020-07-16T06:23:01Z'
2020-07-16 15:18:14 [16] waited idle for 173 cycles.
2020-07-16 15:18:14 [16] write_start of process 818. Considered as reading:.
2020-07-16 15:18:14 [818] write_start() end
2020-07-16 15:34:20 [819] request_read_and_idx() start
2020-07-16 15:34:20 [16] waited idle for 9625 cycles.
[...]
2020-07-16 15:34:56 [16] waited idle for 8 cycles.
2020-07-16 15:34:56 [819] Dispatcher_Client::request_read_and_idx::timeout /osm3s_v0.7.55_osm_base 0 Success 92.250.9.186
2020-07-16 15:34:56 [16] waited idle for 8 cycles.
2020-07-16 15:34:56 [16] hangup of process 819.
2020-07-16 15:40:34 [832] request_read_and_idx() start
2020-07-16 15:40:34 [16] waited idle for 3365 cycles.
[...]
2020-07-16 15:48:28 [16] waited idle for 8 cycles.
2020-07-16 15:48:28 [833] Dispatcher_Client::request_read_and_idx::timeout /osm3s_v0.7.55_osm_base 0 Success 92.250.9.186
2020-07-16 15:48:28 [16] waited idle for 8 cycles.
2020-07-16 15:48:28 [16] hangup of process 833.
2020-07-16 15:50:14 [16] waited idle for 1057 cycles.
2020-07-16 15:50:14 [16] query_my_status by process 836.
2020-07-16 15:50:14 [16] waited idle for 1 cycles.
2020-07-16 15:50:14 [16] hangup of process 836.
2020-07-16 15:51:08 [16] waited idle for 546 cycles.
2020-07-16 15:51:08 [16] query_my_status by process 839.
2020-07-16 15:51:08 [16] waited idle for 1 cycles.
2020-07-16 15:51:08 [16] hangup of process 839.
2020-07-16 15:51:44 [818] write_commit() start 149937 1838062 420189 1134147 270038
2020-07-16 15:51:44 [16] waited idle for 357 cycles.
2020-07-16 15:51:44 [16] write_commit of process 818.
2020-07-16 15:51:44 [818] write_commit() end
2020-07-16 15:51:44 [16] hangup of process 818.
2020-07-16 15:52:22 [1267] write_start() start version='2020-07-16T09:50:01Z'
2020-07-16 15:52:22 [16] waited idle for 388 cycles.
2020-07-16 15:52:22 [16] write_start of process 1267. Considered as reading:.
2020-07-16 15:52:22 [1267] write_start() end

I googled all over the place but cannot find a clue.

Thanks for any pointers!

Failed to process planet file

[ ok ]
osm:~# service docker start

Starting docker ...
[ ok ]
osm:~# docker run \

-e OVERPASS_META=yes \
-e OVERPASS_MODE=init \
-e OVERPASS_PLANET_URL=https://download.geofabrik.de/europe/germany/baden-

wuerttemberg-latest.osm.bz2 \

-e OVERPASS_DIFF_URL=http://download.geofabrik.de/europe/germany/baden-wue

rttemberh-updates/ \

-e OVERPASS_RULES_LOAD=10 \
-v /big/docker/overpass_db/:/db \
-i -t \
--name overpass_bw wiktorn/overpass-api

No database directory. Initializing
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 717M 100 717M 0 0 1708k 0 0:07:09 0:07:09 --:--:-- 1602k
[======================================================================] 100%
Reading XML file ...terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
/app/bin/init_osm3s.sh: line 44: 15 Broken pipe bunzip2 < $PLANET_FE
16 Aborted | $EXEC_DIR/bin/update_database --db-dir=$DB_DIN
Failed to process planet file

It does not look like the update is working

Update finished with status code: 3 INFO: Using replication server at http://download.geofabrik.de/europe/germany/baden-wuerttemberg/tue/ DEBUG: Using given sequence ID 2593 DEBUG: Starting download at ID 2594 (max 100 MB) Empty version, skipping file

Increasing the dispatcher rate limit

The dispatcher server receives an argument called --rate-limit. How to set this up using this docker image?

wiktorn / overpass-api Goto Github PK

overpass-api's Issues

Steps to reproduce:

Recommend Projects

Recommend Topics

Recommend Org