merlinnot / nominatim-docker Goto Github PK

Fully-featured container for geocoding, reverse geocoding and address lookup based on Nominatim and Open Street Map data.

License: MIT License

Shell 20.78% Dockerfile 79.22%

geocoding reverse-geocoding openstreetmap nominatim docker-image

nominatim-docker's Introduction

Nominatim Docker container

Fully-featured container for geocoding, reverse geocoding and address lookup based on Nominatim and Open Street Map data.

Build

The build process is fairly straightforward, but requires lots of computer resources and can take days to complete.

To build, install Docker and run following command:

docker build github.com/merlinnot/nominatim-docker \
  -t nominatim \
  --build-arg BUILD_THREADS=16 \
  --build-arg BUILD_MEMORY=32GB \
  --build-arg OSM2PGSQL_CACHE=28000 \
  --build-arg RUNTIME_THREADS=2 \
  --build-arg RUNTIME_MEMORY=8GB

Full list of build arguments:

Name	Default	Description
BUILD_THREADS	16	Number of threads used during build process.
BUILD_MEMORY	32GB	Memory dedicated to postgresql during build process.
PGSQL_VERSION	9.6	PostgreSQL version.
POSTGIS_VERSION	2.4	PostGIS version.
OSM2PGSQL_CACHE	24000	osm2pgsql cache size in MB, should be set to about 75% of memory available during build process, to a maximum of about 30000. Additional RAM will not be used.
PBF_URL	planet	URL to OpenStreetMap data in PBF format. See geofabrik.de for extracts.
REPLICATION_URL	planet	URL to directory with periodic updates. I recommend using geofabrik.de (see `raw directory index` and lookup `updates`).
IMPORT_ADMINISTRATIVE	false	Data provided in `PBF_FILE` might be used to import only administrative boundaries. Importing only administrative boundaries is much faster and is useful for tasks like country code reverse geocoding.
RUNTIME_THREADS	2	Estimated number of threads available to the running container.
RUNTIME_MEMORY	8GB	Estimated memory size available to the running container.

Run

To run container built in the previous step use

docker run --restart=always -d -p 80:80 merlinnot/nominatim-docker

API will be available at port 80 under /nominatim/ directory.

Develop

This project uses Devver, but feel free to use any of your favorite editors.

For development purposes I strongly encourage to start a build process using URLs for Monacco, it makes the process much faster:

docker build github.com/merlinnot/nominatim-docker \
  -t nominatim \
  --build-arg BUILD_THREADS=2 \
  --build-arg BUILD_MEMORY=8GB \
  --build-arg OSM2PGSQL_CACHE=2000 \
  --build-arg RUNTIME_THREADS=2 \
  --build-arg RUNTIME_MEMORY=8GB \
  --build-arg PBF_URL=http://download.geofabrik.de/europe/monaco-latest.osm.pbf \
  --build-arg REPLICATION_URL=http://download.geofabrik.de/monaco-updates

nominatim-docker's People

Contributors

Stargazers

Watchers

nominatim-docker's Issues

ERROR: No Data during build

osm2pgsql version 0.96.0 (64 bit id space)

Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=12000MB, maxblocks=192000*65536, allocation method=11
Mid: pgsql, cache=12000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Parsing gazetteer style file '/srv/nominatim/Nominatim/settings/import-full.style'.
Using projection SRS 4326 (Latlong)
NOTICE: table "place" does not exist, skipping

Reading in file: /srv/nominatim/src/data.osm.pbf
Using PBF parser.
Processing: Node(358820k 707.7k/s) Way(0k 0.00k/s) Relation(0 0.00/s)string(7) "No Data"
ERROR: No Data
The command '/bin/sh -c service postgresql start && sudo -u nominatim ${USERHOME}/Nominatim/build/utils/setup.php --osm-file /srv/nominatim/src/data.osm.pbf --all --threads ${BUILD_THREADS} --osm2pgsql-cache ${OSM2PGSQL_CACHE} && service postgresql stop' returned a non-zero code: 255

Would it be possible for this to work on weaker hardware?

Would this instance work on a 12GB RAM/i5/1TB HDD laptop for whole planet data?
Would you go against setting it up on so much weaker hardware?

Updating the database

Great solution - thanks!
One problem I have - after deployment into Azure docker container, it is working great, but no updates happening (I used configuration --build-arg REPLICATION_URL=http://download.geofabrik.de/north-america/us-south-updates )

Is there a step I am missing? Cron configuration?

Unable to build nominatim planet

Hi,
Unable to build nominatim planet when docker build process of
Step 35/68 : RUN curl -L ${PBF_URL} --create-dirs -o /srv/nominatim/src/data.osm.pbf
then ran into this issue.
ERROR: Error processing tar file(exit status 1): unexpected EOF

The machine info:
CPU: 8 core
MEM: 32GB
STORAGE: 10TB

Docker version:
Client:
Version: 18.06.0-ce
API version: 1.38
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:11:02 2018
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.06.0-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:09:05 2018
OS/Arch: linux/amd64
Experimental: false

Let me know if you need more info.

Thanks in advance.

Build stopped working

docker build github.com/merlinnot/nominatim-docker   -t nominatim   --build-arg BUILD_THREADS=1   --build-arg BUILD_MEMORY=2GB   --build-arg OSM2PGSQL_CACHE=1000 --build-arg PBF_URL=http://download.geofabrik.de/europe/monaco-latest.osm.pbf   --build-arg REPLICATION_URL=http://download.geofabrik.de/monaco-updates

returns

"$libdir/postgis-2.5": No such file or directory  
ERROR: pgsql returned with error code (3)

did not find all addresses which are found by nominatim.openstreetmap.org

I installed a nominatim server with germany-latest.osrm.pbf

so far so good

with some addresses my server has some problem while the nominatim.openstreetmap.org find the address.

any idea why?

post scriptum: can you also set up a nominatim server using a local merged pbf file?

postgis error

when i launch the script with the monaco example i have this error :

2019-03-08 21:21:21 == module path: /srv/nominatim/Nominatim/build/module
2019-03-08 21:21:21 == Create DB
2019-03-08 21:21:23 == Setup DB
Postgres version found: 9.06
ERROR: could not access file "$libdir/postgis-2.5": No such file or directory

PHP Version Upgrade

Currently the system uses PHP 7.0 which is EOL as of now.

Is it good to upgrade to 7.3?

Build failed on Ubuntu

Tried this command per the documentation:

docker build github.com/merlinnot/nominatim-docker
-t test
--build-arg BUILD_THREADS=16
--build-arg BUILD_MEMORY=16GB
--build-arg OSM2PGSQL_CACHE=28000
--build-arg RUNTIME_THREADS=2
--build-arg RUNTIME_MEMORY=4GB

Ran for a while and then failed with this error on step 49:

Step 49/68 : RUN service postgresql start && sudo -u nominatim ${USERHOME}/Nominatim/build/utils/setup.php --osm-file /srv/nominatim/src/data.osm.pbf --all --threads ${BUILD_THREADS} --osm2pgsql-cache ${OSM2PGSQL_CACHE} && service postgresql stop
---> Running in 4ff1cb61c24b

Starting PostgreSQL 9.6 database server
...done.
2019-06-10 06:40:08 == module path: /srv/nominatim/Nominatim/build/module
2019-06-10 06:40:08 == Create DB
2019-06-10 06:40:09 == Setup DB
Postgres version found: 9.06
ERROR: could not access file "$libdir/postgis-2.5": No such file or directory
ERROR: pgsql returned with error code (3)
string(34) "pgsql returned with error code (3)"
The command '/bin/sh -c service postgresql start && sudo -u nominatim ${USERHOME}/Nominatim/build/utils/setup.php --osm-file /srv/nominatim/src/data.osm.pbf --all --threads ${BUILD_THREADS} --osm2pgsql-cache ${OSM2PGSQL_CACHE} && service postgresql stop' returned a non-zero code: 255

Database query failed Step 49/68

Hi @merlinnot ,

I got the error while building the docker file. Could you help to resolve it? I run with POSTGIS_VERSION=2.5

Best,
Thanh

Package 'postgresql-9.6-postgis-scripts' has no installation candidate

After running:

docker build github.com/merlinnot/nominatim-docker \
  -t nominatim \
  --build-arg BUILD_THREADS=4 \
  --build-arg BUILD_MEMORY=4GB \
  --build-arg OSM2PGSQL_CACHE=28000 \
  --build-arg RUNTIME_THREADS=2 \
  --build-arg RUNTIME_MEMORY=4GB \
  --build-arg PBF_URL="https://download.geofabrik.de/asia/maldives-latest.osm.pbf"

Step 18/68 : RUN apt-get install -y --no-install-recommends       apache2       build-essential       ca-certificates       cmake       curl       g++       git       libapache2-mod-php       libboost-dev       libboost-filesystem-dev       libboost-python-dev       libboost-system-dev       libbz2-dev       libexpat1-dev       libgeos-dev       libgeos++-dev       libpq-dev       libproj-dev       libxml2-dev      openssl       osmosis       php       php-db       php-intl       php-pear       php-pgsql       postgresql-${PGSQL_VERSION}-postgis-${POSTGIS_VERSION}       postgresql-${PGSQL_VERSION}-postgis-scripts       postgresql-contrib-${PGSQL_VERSION}       postgresql-server-dev-${PGSQL_VERSION}       python       python-pip       python-setuptools       sudo       zlib1g-dev
 ---> Running in 2060a7f3f51f
Reading package lists...
Building dependency tree...
Reading state information...
Package postgresql-9.6-postgis-scripts is a virtual package provided by:
  postgresql-9.6-postgis-3-scripts 3.0.0+dfsg-2~exp1.pgdg16.04+1
  postgresql-9.6-postgis-2.5-scripts 2.5.3+dfsg-2.pgdg16.04+1

E: Package 'postgresql-9.6-postgis-scripts' has no installation candidate
The command '/bin/sh -c apt-get install -y --no-install-recommends       apache2       build-essential       ca-certificates       cmake       curl       g++       git       libapache2-mod-php       libboost-dev       libboost-filesystem-dev       libboost-python-dev       libboost-system-dev       libbz2-dev       libexpat1-dev       libgeos-dev       libgeos++-dev       libpq-dev       libproj-dev       libxml2-dev      openssl       osmosis       php       php-db       php-intl       php-pear       php-pgsql       postgresql-${PGSQL_VERSION}-postgis-${POSTGIS_VERSION}       postgresql-${PGSQL_VERSION}-postgis-scripts       postgresql-contrib-${PGSQL_VERSION}       postgresql-server-dev-${PGSQL_VERSION}       python       python-pip       python-setuptools       sudo       zlib1g-dev' returned a non-zero code: 100

Build Failaure

Hi,
We are getting build failures with below error -

Reading in file: /srv/nominatim/src/data.osm.pbf
Using PBF parser.
Processing: Node(2727010k 385.4k/s) Way(0k 0.00k/s) Relation(0 0.00/s)Error executing external command: /srv/nominatim/Nominatim/build/osm2pgsql/osm2pgsql -lsc -O gazetteer --hstore --number-processes 1 -C 28000 -P 5432 -d nominatim /srv/nominatim/src/data.osm.pbf
ERROR: Error executing external command: /srv/nominatim/Nominatim/build/osm2pgsql/osm2pgsql -lsc -O gazetteer --hstore --number-processes 1 -C 28000 -P 5432 -d nominatim /srv/nominatim/src/data.osm.pbf
The command '/bin/sh -c service postgresql start && sudo -u nominatim ${USERHOME}/Nominatim/build/utils/setup.php --osm-file /srv/nominatim/src/data.osm.pbf --all --threads ${BUILD_THREADS} --osm2pgsql-cache ${OSM2PGSQL_CACHE} && service postgresql stop' returned a non-zero code: 255

Note - We are using below machine for the same -

RAM: 32GB
CORE: 8
STORAGE: 1TB

Build config:
BUILD_THREADS=16
BUILD_MEMORY=26GB
OSM2PGSQL_CACHE=20000

Please let me know if you need more info.

Thanks,

PostGIS error in step 49/68

HI, has anyone seen this error before (at end of log)? No other errors encountered before.
I tried twice on a clean Docker droplet :

...
ALTER SYSTEM SET full_page_writes TO 'off';
ALTER SYSTEM
ALTER SYSTEM SET logging_collector TO 'off';
ALTER SYSTEM

Stopping PostgreSQL 9.6 database server
...done.
---> 645a7a0deb5e
Removing intermediate container 18ccc7cca4ea
Step 47/68 : USER root
---> Running in 0e348fc8ab02
---> 586f64ff0789
Removing intermediate container 0e348fc8ab02
Step 48/68 : ARG OSM2PGSQL_CACHE=24000
---> Running in aa3904e144b2
---> 7ac49185f394
Removing intermediate container aa3904e144b2
Step 49/68 : RUN service postgresql start && sudo -u nominatim ${USERHOME}/Nominatim/build/utils/setup.php --osm-file /srv/nominatim/src/data.osm.pbf --all --threads ${BUILD_THREADS} --osm2pgsql-cache ${OSM2PGSQL_CACHE} && service postgresql stop
---> Running in d10f2fa5e72d
Starting PostgreSQL 9.6 database server
...done.
2018-01-16 10:39:01 == WARNING: resetting threads to 2
2018-01-16 10:39:01 == Create DB
2018-01-16 10:39:03 == Setup DB
Postgres version found: 9.6
pgsql returned with error code (3)
ERROR: could not access file "$libdir/postgis-2.4": No such file or directory
ERROR: pgsql returned with error code (3)
The command '/bin/sh -c service postgresql start && sudo -u nominatim ${USERHOME}/Nominatim/build/utils/setup.php --osm-file /srv/nominatim/src/data.osm.pbf --all --threads ${BUILD_THREADS} --osm2pgsql-cache ${OSM2PGSQL_CACHE} && service postgresql stop' returned a non-zero code: 255

Simple question about postgre access

First thank you for sharing this very nice docker project !
I have just a little question : is it possible to connecte to the postgre nominatim schema from outside the container ? (for port it is ok we can map it but for user and password do you have an idea ?)

I need to specify a connection string to photon that's why.

Thank you very much in advance

Feature requirements for v2

I intend to work on a v2 version and I'd love to hear your feedback and brainstorm some ideas.

What I have in mind is:

Separate it into two containers, provide a Docker Compose file:
- Postgres with storage attached as a volume
- An API
Support root path (drop the /nominatim prefix for paths).
Update all dependencies.
Enable automated updates with Renovate if possible.
Resolve an issue with the database updates.
Restructure the documentation to include more details on suggested deployment strategies, elaborate on options, add contributing guidelines etc.

ImportError: No module named 'psycopg2'

/srv/nominatim/Nominatim/nominatim/nominatim.py:307: SyntaxWarning: assertion is always true, perhaps remove parentheses?
assert(False, "Unreachable code")
Traceback (most recent call last):
File "/srv/nominatim/Nominatim/nominatim/nominatim.py", line 31, in
import psycopg2
ImportError: No module named 'psycopg2'
string(33) "error status 1 running nominatim!"
ERROR: error status 1 running nominatim!

E: Package 'postgresql-9.6-postgis-scripts' has no installation candidate

I've ran the build script as mentioned in README, and I get this error:

E: Package 'postgresql-9.6-postgis-scripts' has no installation candidate

A little more details:

Step 17/68 : USER root
 ---> Using cache
 ---> 57e6ed1b9d37
Step 18/68 : RUN apt-get install -y --no-install-recommends       apache2       build-essential       ca-certificates       cmake       curl       g++       git       libapache2-mod-php       libboost-dev       libboost-filesystem-dev       libboost-python-dev       libboost-system-dev       libbz2-dev       libexpat1-dev       libgeos-dev       libgeos++-dev       libpq-dev       libproj-dev       libxml2-dev      openssl       osmosis       php       php-db       php-intl       php-pear       php-pgsql       postgresql-${PGSQL_VERSION}-postgis-${POSTGIS_VERSION}       postgresql-${PGSQL_VERSION}-postgis-scripts       postgresql-contrib-${PGSQL_VERSION}       postgresql-server-dev-${PGSQL_VERSION}       python       python-pip       python-setuptools       sudo       zlib1g-dev
 ---> Running in 3f7ab2d8c047
Reading package lists...
Building dependency tree...
Reading state information...
Package postgresql-9.6-postgis-scripts is a virtual package provided by:
  postgresql-9.6-postgis-3-scripts 3.0.0+dfsg-2~exp1.pgdg16.04+1
  postgresql-9.6-postgis-2.5-scripts 2.5.3+dfsg-2.pgdg16.04+1

E: Package 'postgresql-9.6-postgis-scripts' has no installation candidate
The command '/bin/sh -c apt-get install -y --no-install-recommends       apache2       build-essential       ca-certificates       cmake       curl       g++       git       libapache2-mod-php       libboost-dev       libboost-filesystem-dev       libboost-python-dev       libboost-system-dev       libbz2-dev       libexpat1-dev       libgeos-dev       libgeos++-dev       libpq-dev       libproj-dev       libxml2-dev      openssl       osmosis       php       php-db       php-intl       php-pear       php-pgsql       postgresql-${PGSQL_VERSION}-postgis-${POSTGIS_VERSION}       postgresql-${PGSQL_VERSION}-postgis-scripts       postgresql-contrib-${PGSQL_VERSION}       postgresql-server-dev-${PGSQL_VERSION}       python       python-pip       python-setuptools       sudo       zlib1g-dev' returned a non-zero code: 100

Proper way to run built container?

That's an amazing project, thanks!

I'm having some trouble to figure out the proper way to run the built container.

The recommended command
docker run --restart=always -d -p 80:80 merlinnot/nominatim-docker
fails with the following message:

Unable to find image 'merlinnot/nominatim-docker:latest' locally
docker: Error response from daemon: manifest for merlinnot/nominatim-docker:latest not found.
See 'docker run --help'.

If I do:
sudo docker run --restart=always -d -p 80:80 nominatim
It works, but the base href on HTML is set to the container ID, like: <base href="http://457fa9e123ba/" />

So, there seem to be two problems here:

merlinnot/nominatim-docker name doesn't work?
base href config is broken? CONST_Website_BaseURL should have hostname?

I was able to get around those problems by manually setting the container hostname:
sudo docker run --restart=always -h 222.22.111.111/nominatim -d -p 80:80 nominatim
Perhaps you want to document this?