Problem after importing data about overpass-api HOT 7 CLOSED

You can check out my solution here:

https://github.com/codeformuenster/verkehrsunfaelle/blob/python-importer/docker-compose.yml

Also check out the files in the overpass directory in the same branch / repo

This assumes you want a container for single use.

from overpass-api.

I guess that you're missing OVERPASS_PLANET_SEQUENCE_ID environment variable. You need this, to reflect, how the dump that you're loaded is positioned against minute diff repository. Each diff repository has its own sequence so you need to check the one that you choose for correct number.

Please bear in mind, that though you've loaded only one Land, your diffs will be updating your database with whole planet so you will have partial (but growing) data from other parts of the world.

from overpass-api.

Thanks for your answer, how to find the sequence id though?

As per your suggestion I switched to http://download.openstreetmap.fr/replication/europe/germany/nordrhein_westfalen/munster/minute/ for the diff_url and 3397105 (which is the sequenceNumber from this file) for the sequence id.

Or is there a way to disable the updates altogether? I only need the overpass container for bulk queries. I tried to comment out the portions [program:fetch_diff], [program:apply_diff], [program:dispatcher_areas] and [program:areas_rules] in supervisord.conf, then mounting the edited file into the container with -v, but sadly this resulted in almost the same error.

from overpass-api.

I didn't thought about running instance that's not updating itself, so this is something that's missing.

As for sequenceNumber I usually do this way, that I check the timestamp of planet file, then take sequence number that's a bit before this date. To find the sequence number, I navigate the directories in minute diffs, to find file that was modified at the data that I'm looking for.

So for example, If I have a dump file from 2019-02-23 04:00:00 am, I'd look for sequence file from midnight that day (using modified date), and it is this file. In the contents you'll have the sequence number.

It's good to have a sequence number not later than your dump so you'll not loose any objects that were created/updated between your planet dump file and since you're starting synchronization. Overlapping is not a problem, as the data will easily converge to current state.

Did it work properly when you set the sequence id?

from overpass-api.

Sorry for the late reply. I found out some stuff

It seems the --meta stuff does not work as expected. When setting it to no, the script dispatcher_start.sh does not work anymore. The empty string is being parsed as parameter which is then unknown to the actual dispatcher binary.

Also omitting the meta environment variable completely should have disabled the meta stuff but .meta files are generated nevertheless ?!

I have now done the following:

my supervisord.conf:

[supervisord]
nodaemon=true

[program:overpass_dispatch]
command=/bin/bash -c "find /db/db -type s -print0 | xargs -0 --no-run-if-empty rm && /app/bin/dispatcher --osm-base --db-dir=/db/db"
user=overpass
redirect_stderr=true
priority=1

[program:nginx]
command=nginx
priority=2

[program:fcgiwrap]
command=/bin/bash -c "find /nginx -type s -print0 | xargs -0 --no-run-if-empty rm && fcgiwrap -s unix:/nginx/fcgiwrap.socket"
redirect_stderr=true
user=nginx
priority=3

My docker-entrypoint.sh:

#!/bin/bash

set -eo pipefail
shopt -s nullglob
OVERPASS_COMPRESSION=${OVERPASS_COMPRESSION:-gz}

if [ ! -d /db/db ] ; then
  lftp -c "get -c \"$OVERPASS_PLANET_URL\" -o /db/planet; exit" \
  && /app/bin/init_osm3s.sh /db/planet /db/db /app "--compression-method=$OVERPASS_COMPRESSION" \
  && rm /db/planet \
  && cp -r /app/etc/rules /db/db \
  && chown -R overpass:overpass /db \
  && echo "Overpass ready, you can start your container with docker start"
  exit
fi

exec /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

from overpass-api.

I am struggling with the same issue, this is my docker-compose.yml

version: '3'
services:
  api:
    image: wiktorn/overpass-api:latest
    container_name: overpass-api
    restart: always
    ports:
      - '12345:80'
    environment:
      - OVERPASS_META=yes
      - OVERPASS_MODE=init
      - OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/spain-latest.osm.bz2
      - OVERPASS_DIFF_URL=http://download.openstreetmap.fr/replication/europe/spain/minute/
      - OVERPASS_PLANET_SEQUENCE_ID=3370000
    volumes:
      - /Workspace/docker/overpass_db/:/db
    stdin_open: true
    tty: true

I followed the steps to get the PLANET_SEQUENCE_ID and it didn't work, also tried with different META config and nothing, any idea to start investigating from?

from overpass-api.

It worked 🎉
A single use container is ok, thanks a lot @ubergesundheit !

from overpass-api.