18 THREADS seems too high, that'd result in up to (3 + (1+18) * 14) = 269 connections, whereas max_connections in postgresql.custom.conf.tmpl is configured to 250.

Also resulting from this: does your system really have 269 * 128 MB + 8192 MB = 42 GB of memory?

See #330

from openstreetmap-tile-server.

Thank you so much for the reply! Yes, this system has an Intel i9-10900X (20 threads) and 64 GB of RAM. I'll try again with less threads.

Best,
Eric

from openstreetmap-tile-server.

Well, I got a little further. Now it says I ran out of disk space. The osm-data volume resides on an external USB3-to-SATA adapter with a 1.9TB SSD. With everything idle now (after import aborted), the volume says it is 926 GB and windows says the disk (e:) has 760 GB free. My boot drive (c:) has 500 GB free. Any assistance would be greatly appreciated!

tech@4YF0GT3:/mnt/c/Users/ehughes$ docker run
-v /mnt/e/planet-231002.osm.pbf:/data/region.osm.pbf
-v osm-data:/data/database/
-e "FLAT_NODES=enabled"
-e "OSM2PGSQL_EXTRA_ARGS=-C 8192"
-e THREADS=10
overv/openstreetmap-tile-server
import

• '[' import == import ']'
• mkdir -p /data/database/postgres/
• chown renderer: /data/database/
• chown -R postgres: /var/lib/postgresql /data/database/postgres/
• '[' '!' -f /data/database/postgres/PG_VERSION ']'
• createPostgresConfig
• cp /etc/postgresql/15/main/postgresql.custom.conf.tmpl /etc/postgresql/15/main/conf.d/postgresql.custom.conf
• sudo -u postgres echo 'autovacuum = on'
• cat /etc/postgresql/15/main/conf.d/postgresql.custom.conf

Suggested minimal settings from

https://ircama.github.io/osm-carto-tutorials/tile-server-ubuntu/

shared_buffers = 128MB
min_wal_size = 1GB

max_wal_size = 2GB # Overridden below

maintenance_work_mem = 256MB

Suggested settings from

https://github.com/openstreetmap/chef/blob/master/roles/tile.rb#L38-L45

max_connections = 250
temp_buffers = 32MB
work_mem = 128MB
wal_buffers = 1024kB
wal_writer_delay = 500ms
commit_delay = 10000

checkpoint_segments = 60 # unrecognized in psql 10.7.1

max_wal_size = 2880MB
random_page_cost = 1.1
track_activity_query_size = 16384
autovacuum_vacuum_scale_factor = 0.05
autovacuum_analyze_scale_factor = 0.02

listen_addresses = '*'
autovacuum = on

• service postgresql start

• Starting PostgreSQL 15 database server
  ...done.

• sudo -u postgres createuser renderer
• sudo -u postgres createdb -E UTF8 -O renderer gis
• sudo -u postgres psql -d gis -c 'CREATE EXTENSION postgis;'
  CREATE EXTENSION
• sudo -u postgres psql -d gis -c 'CREATE EXTENSION hstore;'
  CREATE EXTENSION
• sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO renderer;'
  ALTER TABLE
• sudo -u postgres psql -d gis -c 'ALTER TABLE spatial_ref_sys OWNER TO renderer;'
  ALTER TABLE
• setPostgresPassword
• sudo -u postgres psql -c 'ALTER USER renderer PASSWORD '''renderer''''
  ALTER ROLE
• '[' '!' -f /data/region.osm.pbf ']'
• '[' -n '' ']'
• '[' disabled == enabled ']'
• '[' disabled == 1 ']'
• '[' -f /data/region.poly ']'
• '[' enabled == enabled ']'
• OSM2PGSQL_EXTRA_ARGS='-C 8192 --flat-nodes /data/database/flat_nodes.bin'
• sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /data/style/openstreetmap-carto.lua --number-processes 10 -S /data/style/openstreetmap-carto.style /data/region.osm.pbf -C 8192 --flat-nodes /data/database/flat_nodes.bin
  2023-10-26 18:54:07 osm2pgsql version 1.6.0
  2023-10-26 18:54:07 Database version: 15.3 (Ubuntu 15.3-1.pgdg22.04+1)
  2023-10-26 18:54:07 PostGIS version: 3.3
  2023-10-26 18:54:08 Setting up table 'planet_osm_point'
  2023-10-26 18:54:08 Setting up table 'planet_osm_line'
  2023-10-26 18:54:08 Setting up table 'planet_osm_polygon'
  2023-10-26 18:54:08 Setting up table 'planet_osm_roads'
  2023-10-29 18:59:03 Reading input files done in 259495s (72h 4m 55s).
  2023-10-29 18:59:03 Processed 8658483163 nodes in 3478s (57m 58s) - 2490k/s
  2023-10-29 18:59:03 Processed 971692934 ways in 175068s (48h 37m 48s) - 6k/s
  2023-10-29 18:59:03 Processed 11340554 relations in 80949s (22h 29m 9s) - 140/s
  2023-10-29 18:59:03 Clustering table 'planet_osm_polygon' by geometry...
  2023-10-29 18:59:03 Clustering table 'planet_osm_roads' by geometry...
  2023-10-29 18:59:03 Clustering table 'planet_osm_line' by geometry...
  2023-10-29 18:59:03 Clustering table 'planet_osm_point' by geometry...
  2023-10-29 18:59:06 Done postprocessing on table 'planet_osm_nodes' in 0s
  2023-10-29 18:59:06 Building index on table 'planet_osm_rels'
  2023-10-29 18:59:06 Building index on table 'planet_osm_ways'
  2023-10-29 22:22:17 ERROR: Database error: ERROR: could not extend file "base/16385/7440649.3": No space left on device
  HINT: Check free disk space.

from openstreetmap-tile-server.

If I remember correctly, the docker settings on Windows might impose a maximum volume size limit? IDK.

Likely it cleaned up the files from the step it was at after the error happened and that is why there is more free space now than when the error occured. According to the logs the data import took 3 days, and it failed 3 hours later when creating the indicies for the planet_osm_ways table.

According to https://wiki.openstreetmap.org/wiki/Planet.osm the uncompressed data is 1.8 TB, so 1.9 TB for the whole planet might not be enough? But the import doesn't import all the data, so it should be less, though the database also needs space for its own and for indicies and stuff. I think 1.9 TB should be enough if there is nothing else on the disk. What I read from about two years ago the size was about 1.2 TB imported.

from openstreetmap-tile-server.

Just to follow-up, the issue was indeed Windows (and/or docker) imposing a volume size limit of 1TB. I tried all sorts of things to increase it, but nothing worked. I finally gave up and moved to Ubuntu and everything worked. Really appreciate your help!!

from openstreetmap-tile-server.