Tried to install following the directions at http://wiki.openstreetmap.org/wiki/Osm2pgsql#With_Homebrew but got the following error:

Error: No available formula for protobuf-c

On a Debian Squeeze box, the master branch of osm2pgsql imports everything as expected from a planet.osm file. However, the same repository on an ubuntu 12.04 install misses some polygon features like forests and the rest of a river after a damn, some islands etc.

I tested osm2pgsql on a debian box, pointing at the database on my new ubuntu machine. So, it's not the postgres server, but probably one of the libraries that osm2pgsql depends on for parsing. Same problem happens on the PBF or XML file.

Where do I start to get this figured out? I have a new shiny new dedicated tile server and need to migrate off the old server. Due to dependency hell I have not been able to get the exact versions of libgeos and postgis running on the new ubuntu server.

Originally the --number-processes option defaulted to one because the multi-threaded code wasn't as well tested. At this point it should be well tested and unless they have a reason to do otherwise they should use the number of CPUs they have, particularly for an import. Larger servers might want to use less but they should be tuning their own osm2pgsql anyways.

The default for --number-processes should be the number of cores.

Relations with multiple members like http://www.openstreetmap.org/?lat=57.011&lon=-6.428&zoom=10&layers=M are rendered multiple times and are a bit of a cartographic disaster.

One possible fix is to import with osm2pgsql in -G mode to create a MULTIPOLYGON instead of multiple POLYGONs. @springmeyer informs me that mapnik should render the label on just the largest POLYGON within the MULTIPOLYGON by default.

Do we know anyone running with -G?

time bzcat belgium-latest.osm.bz2 | osm2pgsql -r libxml2 -d osm2pgsqltest -l -O null - takes 1m21s but time bzcat belgium-latest.osm.bz2 | osm2pgsql -r primitive -d osm2pgsqltest -l -O null - takes 0.2s, which is clearly absurd. It also only parses 30k nodes.

When removing -O null, the resulting database is under 1MB.

If I bunzip2 to disk and give it the file, primitive does work and takes 46s (-O null) to libxml2's 71s. The pbf takes about 7.25 seconds.

If we aren't maintaining the primitive parser, I'd suggest removing it. PBF is absurdly faster than either XML parser for where speed matters.

Hello again :),

so now i tried to import oberbayern-latest.osm.bz2 into my db using the latest version of osm2pgsql (with fix from #31), and now i have the polygon called "Stadtbezirk 04 Schwabing-West". Thank you very much for the fix. But now the problem occured again. So what did i do:

First i filtered out admin boundaries using osmosis:

bzcat oberbayern-latest.osm.bz2 | ./osmosis/bin/osmosis --read-xml enableDateParsing=no file=-  --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --write-xml file=-  | bzip2 >  oberbayern-adminbounds.xml.bz2

I used: Osmosis Version 0.43.1

After it i checked if the relation is inside the file "oberbayern-adminbounds.xml.bz2" and found this:

  <relation id="56389" version="25" timestamp="2013-05-31T16:52:28Z" uid="130472" user="fx99" changeset="14290051">
    <member type="way" ref="28823678" role="outer"/>
    <member type="way" ref="30033834" role="outer"/>
    <member type="way" ref="195085889" role="outer"/>
    <member type="way" ref="196473438" role="outer"/>
    <member type="way" ref="194705249" role="outer"/>
    <member type="way" ref="195085890" role="outer"/>
    <member type="way" ref="69685291" role="outer"/>
    <member type="way" ref="69685244" role="outer"/>
    <member type="way" ref="30101361" role="outer"/>
    <member type="way" ref="196473436" role="outer"/>
    <member type="way" ref="45048519" role="outer"/>
    <member type="way" ref="194705245" role="outer"/>
    <member type="way" ref="196167289" role="outer"/>
    <member type="way" ref="196473435" role="outer"/>
    <member type="way" ref="42941597" role="outer"/>
    <member type="way" ref="30033065" role="outer"/>
    <member type="way" ref="30033066" role="outer"/>
    <member type="way" ref="196328310" role="outer"/>
    <member type="way" ref="196328308" role="outer"/>
    <member type="relation" ref="66998" role=""/>
    <member type="relation" ref="66999" role=""/>
    <member type="relation" ref="67000" role=""/>
    <tag k="admin_level" v="9"/>
    <tag k="boundary" v="administrative"/>
    <tag k="is_in" v="München,Bayern,Bundesrepublik Deutschland,Europe"/>
    <tag k="name" v="Stadtbezirk 04 Schwabing-West"/>
    <tag k="note" v="München, Stadtbezirk 4, 3 Bezirksteile"/>
    <tag k="type" v="multipolygon"/>
  </relation>

Then i migrate it to my db using this:

./osm2pgsql/osm2pgsql --cache 6000 --number-processes 7 -S ./default.style --latlong -H localhost  -U michael -P 9201 -d osm oberbayern-adminbounds.xml.bz2

Now i check the db and Schwabing is not there anymore. I checked the db in several ways, but exemplary i show you a short shell dialog:

osm=# SELECT * from planet_osm_polygon where name = 'Stadtbezirk 04 Schwabing-West';
 osm_id | access | addr:housename | addr:housenumber | addr:interpolation | admin_level | aerialway | aeroway | amenity | area | barrier | bicycle | brand | bridge | boundary | building | construction | covered | culvert | cutting | denomination | disused | embankment | foot | generator:source | harbour | highway | historic | horse | intermittent | junction | landuse | layer | leisure | lock | man_made | military | motorcar | name | natural | office | oneway | operator | place | population | power | power_source | public_transport | railway | ref | religion | route | service | shop | sport | surface | toll | tourism | tower:type | tracktype | tunnel | water | waterway | wetland | width | wood | z_order | way_area | timezone | way 
--------+--------+----------------+------------------+--------------------+-------------+-----------+---------+---------+------+---------+---------+-------+--------+----------+----------+--------------+---------+---------+---------+--------------+---------+------------+------+------------------+---------+---------+----------+-------+--------------+----------+---------+-------+---------+------+----------+----------+----------+------+---------+--------+--------+----------+-------+------------+-------+--------------+------------------+---------+-----+----------+-------+---------+------+-------+---------+------+---------+------------+-----------+--------+-------+----------+---------+-------+------+---------+----------+----------+-----
(0 rows)

Thanks very much for your help. You guys are really very responsive, i appreciate this.

PS: Regarding the default.style i just added a timezone line. Now it looks like this:

# This is the style file that matches the old version of osm2pgsql, which
# did not make distinctions between tags for nodes and for ways. There are a
# number of optimisations that can be applied here. Firstly, certain tags
# only apply to only nodes or only ways. By fixing this we reduce the amount
# of useless data loaded into the DB, which is a good thing. Possible
# optimisations for the future:

#1. Generate this file directly from the mapnik XML config, so it's always
# optimal

#2. Extend it so it can understand that highway=tertiary is for ways and
# highway=bus_stop is for nodes

# Flags field isn't used much yet, expect if it contains the text "polygon"
# it indicates the shape is candidate for the polygon table. In the future I
# would like to be able to add directives like "nocache" which tells
# osm2pgsql that it is unlikely this node will be used by a way and so it
# doesn't need to be stored (eg coastline nodes). While in essence an
# optimisation hack, for --slim mode it doesn't matter if you're wrong, but
# in non-slim you might break something!

# Also possibly an ignore flag, for things like "note" and "source" which
# can simply be deleted. (In slim mode this is, does not apply to non-slim
# obviously)

# OsmType  Tag          DataType     Flags
node,way   note         text         delete   # These tags can be long but are useless for rendering
node,way   source       text         delete   # This indicates that we shouldn't store them
node,way   created_by   text         delete

#node,way   note         text         polygon   # These tags can be long but are useless for rendering
#node,way   source       text         polygon   # This indicates that we shouldn't store them
#node,way   created_by   text         polygon

node,way   access       text         linear
node,way   addr:housename      text  linear
node,way   addr:housenumber    text  linear
node,way   addr:interpolation  text  linear 
node,way   admin_level  text         linear
node,way   aerialway    text         linear
node,way   aeroway      text         polygon
node,way   amenity      text         nocache,polygon
node,way   area         text         # hard coded support for area=1/yes => polygon is in osm2pgsql
node,way   barrier      text         linear
node,way   bicycle      text         nocache
node,way   brand        text         linear
node,way   bridge       text         linear
node,way   boundary     text         linear
node,way   building     text         polygon
node       capital      text         linear
node,way   construction text         linear
node,way   covered      text         linear
node,way   culvert      text         linear
node,way   cutting      text         linear
node,way   denomination text         linear
node,way   disused      text         linear
node       ele          text         linear
node,way   embankment   text         linear
node,way   foot         text         linear
node,way   generator:source    text  linear
node,way   harbour      text         polygon
node,way   highway      text         linear
node,way   historic     text         polygon
node,way   horse        text         linear
node,way   intermittent text         linear
node,way   junction     text         linear
node,way   landuse      text         polygon
node,way   layer        text         linear
node,way   leisure      text         polygon
node,way   lock         text         linear
node,way   man_made     text         polygon
node,way   military     text         polygon
node,way   motorcar     text         linear
node,way   name         text         linear
node,way   natural      text         polygon  # natural=coastline tags are discarded by a hard coded rule in osm2pgsql
node,way   office       text         polygon
node,way   oneway       text         linear
node,way   operator     text         linear
node,way   place        text         polygon
node       poi          text
node,way   population   text         linear
node,way   power        text         polygon
node,way   power_source text         linear
node,way   public_transport text     polygon
node,way   railway      text         linear
node,way   ref          text         linear
node,way   religion     text         nocache
node,way   route        text         linear
node,way   service      text         linear
node,way   shop         text         polygon
node,way   sport        text         polygon
node,way   surface      text         linear
node,way   toll         text         linear
node,way   tourism      text         polygon
node,way   tower:type   text         linear
way        tracktype    text         linear
node,way   tunnel       text         linear
node,way   water        text         polygon
node,way   waterway     text         polygon
node,way   wetland      text         polygon
node,way   width        text         linear
node,way   wood         text         linear
node,way   z_order      int4         linear # This is calculated during import
way        way_area     real                # This is calculated during import

# mic specific
node,way   timezone     text         polygon

# If you're interested in bicycle routes, you may want the following fields
# To make these work you need slim mode or the necessary data won't be remembered.
#way       lcn_ref      text     linear
#way       rcn_ref      text     linear
#way       ncn_ref      text     linear
#way       lcn          text     linear
#way       rcn          text     linear
#way       ncn          text     linear
#way       lwn_ref      text     linear
#way       rwn_ref      text     linear
#way       nwn_ref          text     linear
#way       lwn              text     linear
#way       rwn              text     linear
#way       nwn              text     linear
#way       route_pref_color text     linear
#way       route_name       text     linear

# The following entries can be used with the --extra-attributes option
# to include the username, userid, version & timstamp in the DB
#node,way  osm_user       text
#node,way  osm_uid        text
#node,way  osm_version    text
#node,way  osm_timestamp  text

I want to generate line for type=waterway in addition to already existing types. I have added the corresponding code to style.lua, but was blocked by the following code:

  /* Only a limited subset of type= is supported, ignore other */
  if ( (strcmp(type, "route") != 0) && (strcmp(type, "multipolygon") != 0) && (strcmp(type, "boundary") != 0))
    return 0;

This is already handled by style.lua, but it doesn't seem to be present in default tagtransform.

Currently osm2pgsql will create a planet_osm_nodes table even if --flat-nodes is specified. It would probably be best to not create this table.

It is also probably possible to screw up the DB by omitting --flat-nodes on diff processing and this could eliminate that possibility,

When a multipolygon is deleted by diff updates, its polygon is correctly deleted from planet_osm_polygon, but if the outer member ways are still there in the DB, their polygons are not recreated and thus missing after the update because they were not created because of the multipolygon.

It looks like delete_rels query may set pending back true on members to workaround this, but I'm not familiar enough with osm2pgsql code to offer a patch.

The Readme says:

Depending on the command-line switches you can select which projection you
want the database in. You have three choices:

4326: The standard lat/long coordinates
900913: The spherical Mercator projection, used by TileCache, Google Earth etc.
3395: The legacy (broken) WGS84 Mercator projection

It is unclear why EPSG:3395 is considered "broken" relative to EPSG:4326. One should clarify that point.

One should clarify this point.

When importing with --drop or without --slim the resulting tables will be static (i.e. not updated in the future).

The FILLFACTOR parameter allows the creation of physically smaller indices. For static tables fillfactor 100 is best to minimize the physical size. This results in more page splits on updates, but if the tables are never updated this is not an issue.

Using a fillfactor of 100 if --drop is specified or --slim is not specified would reduce index size, slightly speeding up the database.

fillfactor appears to be backwards compatible to postgres 8.2

i have a problem when trying to compile unde Centos 6.3:

$ ./autogen.sh
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I m4
/usr/bin/m4:configure.ac:37: recursion limit of 1024 exceeded, use -L to change it
autom4te: /usr/bin/m4 failed with exit status: 1
aclocal: autom4te failed with exit status: 1
autoreconf: aclocal failed with exit status: 1

$ uname -rvmpio
2.6.32-279.19.1.el6.x86_64 #1 SMP Wed Dec 19 07:05:20 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

$ aclocal --version
aclocal (GNU automake) 1.11.1

$ autoconf --version
autoconf (GNU Autoconf) 2.63

$ m4 --version
m4 (GNU M4) 1.4.13

below there are the required libraries installed

$ rpm -qa | grep -i boost
boost-date-time-1.41.0-11.el6_1.2.x86_64
boost-wave-1.41.0-11.el6_1.2.x86_64
boost-test-1.41.0-11.el6_1.2.x86_64
boost-thread-1.41.0-11.el6_1.2.x86_64
boost-filesystem-1.41.0-11.el6_1.2.x86_64
boost-graph-1.41.0-11.el6_1.2.x86_64
boost-python-1.41.0-11.el6_1.2.x86_64
boost-serialization-1.41.0-11.el6_1.2.x86_64
boost-iostreams-1.41.0-11.el6_1.2.x86_64
boost-1.41.0-11.el6_1.2.x86_64
boost-system-1.41.0-11.el6_1.2.x86_64
boost-regex-1.41.0-11.el6_1.2.x86_64
boost-signals-1.41.0-11.el6_1.2.x86_64
boost-program-options-1.41.0-11.el6_1.2.x86_64
boost-devel-1.41.0-11.el6_1.2.x86_64

$ rpm -qa | grep libxml
libxml2-devel-2.7.6-8.el6_3.4.x86_64
libxml2-2.7.6-8.el6_3.4.x86_64

$ rpm -qa | grep proj
proj-4.8.0-2.rhel6.x86_64
proj-devel-4.8.0-2.rhel6.x86_64

$ rpm -qa | grep geos
geos-3.3.6-4.rhel6.x86_64
geos-devel-3.3.6-4.rhel6.x86_64

$ rpm -qa | grep bzip2
bzip2-libs-1.0.5-7.el6_0.x86_64
bzip2-devel-1.0.5-7.el6_0.x86_64
bzip2-1.0.5-7.el6_0.x86_64

$ rpm -qa | grep zlib
zlib-1.2.3-27.el6.x86_64
zlib-devel-1.2.3-27.el6.x86_64

$ rpm -qa | grep postgres
postgresql92-devel-9.2.2-1PGDG.rhel6.x86_64
postgresql92-9.2.2-1PGDG.rhel6.x86_64
postgresql92-server-9.2.2-1PGDG.rhel6.x86_64
postgresql92-libs-9.2.2-1PGDG.rhel6.x86_64
postgresql92-contrib-9.2.2-1PGDG.rhel6.x86_64

$ rpm -qa | grep gis
postgis2_92-2.0.2-1.rhel6.x86_64

$ rpm -qa | grep -i gettext
gettext-devel-0.17-16.el6.x86_64
gettext-0.17-16.el6.x86_64
gettext-libs-0.17-16.el6.x86_64

Tried to import OSM data into local postgresql database. Excuted
./osm2pgsql -S default.style --slim -d gis -C 2048 --number-processes=1 --cache-strategy=dense japan-latest.osm.bz2

It went well at the first but was broke with an error after 1hr or so. I'm not sure if this is a bug due to my computer environment or something else. Just report it here, hoping someone can shed a light on this.

And the logcat goes like:
Reading in file: ../japan-latest.osm.bz2
Processing: Node(24210k 51.5k/s) Way(0k 0.00k/s) Relation(0 0.00/s)WARNING: Found Out of order node 1052693513 (34582453,9) - this will impact the cache efficiency
Processing: Node(93240k 56.6k/s) Way(0k 0.00k/s) Relation(0 0.00/s)get_node_list failed: ERROR: prepared statement "get_node_list" does not exist
(7)
Arguments were: {31236733,621545916,621545917,31236732,31236568,31236567,31236566,621545902,31236565,621545903,31236564,621545901,621545904,31236563,621545900,31236562,621545905,31236558,31236583,31236584,573323586,31236585,31236586,31236587,31236588,31236589,31236590,31236591,31236629,570755435,621545906,570755436,621545907,31236630,31236631,31236632,31236633,621545908,31236634,31236635,621545909,31236636,31236637,31252890,31252891,31252892},
Error occurred, cleaning up

And the system environment is like:
OS - Ubuntu 12.04
Disk - 40GB
RAM - 2GB
PostgreSQL - postgresql-9.1-postgis, postgresql-contrib-9.1

I'm running postgis 2.0.1, psql 9.1.9, master of osm2pgsql, ubuntu 12.04, 64bit.

Note this problem does not occur for people using 0.81.

SELECT landuse, building FROM planet_osm_polygon WHERE osm_id=-2901483;
SELECT landuse, building FROM planet_osm_polygon WHERE osm_id=-2915918;
SELECT landuse, building FROM planet_osm_polygon WHERE osm_id=-2934273;

First query returned landuse=residential,
2nd query, blank on both,
both were expected behaviors, but in this third query,
for this relation, I received a landuse=residential and building=yes
even though this relation was a type=multipolygon relation.

Building=yes is attributed for all polygons in this relation although these member areas are not all buildings.

I realize these relations, created for the HOT task manager, are hackish, but they may be other use cases where this happens.

Fork() isn't supported by the Windows API:

74608f2#middle-pgsql.c

osm2pgsql cannot build in the usual MingW environment without reverting back the changes which added fork for parallel processing.

middle-pgsql.o: In function `pgsql_iterate_relations':
c:\projects\tools\osminabox1.0_win\openstreetmap-osm2pgsql/middle-pgsql.c:1098:
undefined reference to `fork'
c:\projects\tools\osminabox1.0_win\openstreetmap-osm2pgsql/middle-pgsql.c:1184:
undefined reference to `wait'
middle-pgsql.o: In function `pgsql_iterate_ways':
c:\projects\tools\osminabox1.0_win\openstreetmap-osm2pgsql/middle-pgsql.c:782: u
ndefined reference to `fork'
c:\projects\tools\osminabox1.0_win\openstreetmap-osm2pgsql/middle-pgsql.c:893: u
ndefined reference to `wait'

Might want to consider using fork/exec so the Posix call can be easily replaced by spawn on windows:

http://en.wikipedia.org/wiki/Fork-exec

Microsoft Windows does not support the fork-exec model, as it does not have a system call 
analogous to fork() The spawn() family of functions declared in process.h can replace it in 
cases where the call to fork() is followed directly by exec().

http://stackoverflow.com/questions/12059143/unable-to-code-with-threading-as-an-alternative-to-fork

http://msdn.microsoft.com/en-us/library/y23kc048%28VS.80%29.aspx

One of the largest areas of difference is in the process model. UNIX has fork; Win32 does not. 
Depending on the use of fork and the code base, Win32 has two APIs that can be used: 
CreateProcess and CreateThread. A UNIX application that forks multiple copies of itself 
can be reworked in Win32 to have either multiple processes or a single process with 
multiple threads. If multiple processes are used, there are multiple methods of IPC that 
can be used to communicate between the processes (and perhaps to update the code 
and data of the new process to be like the parent, if the functionality that fork provides is 
needed).

Hi together,
I'd like to suggest to put a hint to the scripts /usr/share/postgresql/9.1/contrib/postgis-1.5/postgis.sql and /usr/share/postgresql/9.1/contrib/postgis-1.5/spatial_ref_sys.sql on the man page of osm2pgsql. I'm sure that it saves a lot of time for some people (besides me ;)).

All the best,
Kalle

Hi, when trying to

git clone https://github.com/openstreetmap/osm2pgsql.git
cd osm2pgsql
./autogen.sh
./configure --with-proj=/opt/local
make

I get the following error:

node-persistent-cache.c: In function 'wait_for_outstanding_io':
node-persistent-cache.c:62:13: warning: implicit declaration of function 'aio_error64' [-Wimplicit-function-declaration]
node-persistent-cache.c:89:17: warning: implicit declaration of function 'aio_suspend64' [-Wimplicit-function-declaration]
node-persistent-cache.c: In function 'writeout_dirty_nodes':
node-persistent-cache.c:112:9: warning: implicit declaration of function 'lseek64' [-Wimplicit-function-declaration]
node-persistent-cache.c: In function 'persistent_cache_nodes_prefetch_async':
node-persistent-cache.c:295:28: error: invalid application of 'sizeof' to incomplete type 'struct aiocb64'
node-persistent-cache.c:296:49: error: dereferencing pointer to incomplete type
node-persistent-cache.c:297:49: error: dereferencing pointer to incomplete type
node-persistent-cache.c:301:49: error: dereferencing pointer to incomplete type
node-persistent-cache.c:303:49: error: dereferencing pointer to incomplete type
node-persistent-cache.c:305:49: error: dereferencing pointer to incomplete type
node-persistent-cache.c:307:13: warning: implicit declaration of function 'aio_write64' [-Wimplicit-function-declaration]
node-persistent-cache.c:318:62: error: invalid application of 'sizeof' to incomplete type 'struct aiocb64'
node-persistent-cache.c:320:45: error: dereferencing pointer to incomplete type
node-persistent-cache.c:321:45: error: dereferencing pointer to incomplete type
node-persistent-cache.c:324:45: error: dereferencing pointer to incomplete type
node-persistent-cache.c:326:45: error: dereferencing pointer to incomplete type
node-persistent-cache.c:328:45: error: dereferencing pointer to incomplete type
node-persistent-cache.c:331:9: warning: implicit declaration of function 'aio_read64' [-Wimplicit-function-declaration]
node-persistent-cache.c: In function 'init_node_persistent_cache':
node-persistent-cache.c:651:13: warning: implicit declaration of function 'posix_fallocate' [-Wimplicit-function-declaration]
make[2]: *** [node-persistent-cache.o] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

Any help would be greatly appreciated.
Stefan

I'm trying to create a nominatim server and attempting to load osm file using osm2pgsql gives segfault as soon as it gets to a 64bit id.

You can get a copy of the small (~15k) datafile from http://www.4x4falcon.com/osm_data/node_0002.osm which shows the problem.

The version of osm2pgsql I'm using is the latest (1 May 2013) from git.

The problem appears to be in the function ram_cache_nodes_set_dense in file node-ram-cache.c

Command to load using nominatim is

./utils/setup.php --osm-file node_0002.osm --all

which then uses osm2pgsql with the following command

./osm2pgsql -lsc -O gazetteer --hstore -C 4250 -d nominatim node_0002.osm

Even if i enter the osm2pgsql command directly osm2pgsql still segfaults.

Running on ubuntu 12.04 lts

Cheers
Ross

parse-pbf.c uses mostly long int to locally store OSM ids. That fails on 32bit systems for 64bit ids. It should be using the explicit osmid_t instead.

Already changed for dense nodes in bbe0de0, rest of code needs to be adapted as well.

I volunteer to build/provide binary packages of osm2pgsql for Mac OS X. My motivation is to make introductory tutorials for those new to OSM as easy as possible to follow (like this one).

But for me to be able to do this I need:

• An official release, or at least a http://semver.org compatible tag
• An agreed upon place to put the binaries that people can trust linking to long into the future from around the web.

Questions? Thoughts?

Why not homebrew for OS X? Because osm2pgsql was kicked out of homebrew for always being broken. It was always broken because the formula pulled from svn head because of lacking releases. So, while homebrew would be the preferred method I think starting with binary packages is a good first step. If osm2pgsql, hopefully through @apmon's leadership, starts providing frequent stable, tagged releases then maybe we, as a community, could consider re-submitting a formula to homebrew.

In addition to OS X (#15) the osm2pgsql community of developers should provide binaries for Windows.

Let's coordinate how to get this done via this ticket.

Tasks:

• - code changes needed to support compiling under mingw, msvc, and/or cygwin
• - volunteer(s) to create binaries on windows and to test them
• - writing an easy installer using nsis that should produce a working executable without any additional installs or development toolkits and should work on 32/64 bit systems ranging from windows XP -> Win8.

Background discussions: hotosm/tech#1

When i try to import data from the following file:
http://download.geofabrik.de/europe/germany/bayern/oberbayern-latest.osm.bz2
(not very large 144MB, so you could try)
i don't get the (for me very import) administrative boundaries in the polygons table (perhaps there are one or two meaningless ones).

Example: There is no "Stadtbezirk 04 Schwabing-West" polygon.
But even the Boundary of the capital München is missing. All the administrative boundaries are missing. I did not check for other polygons or other database files yet, but i try to make a concrete example here, that you could trace it down.

I still had an old database, and there is everything well imported. I checked the ids of the missing polygons and there was something interesting. all these polygons had negative ids in my old database. In the xml file there were only positive ids (i guess you know this well, but this is all new for me). The ids of the xml file were the same like in the database but just positive ones.

I checked the xml file and i seem to miss this one (next to all the others):

<relation id="56389" version="25" timestamp="2012-12-16T09:21:11Z" changeset="14290051" uid="130472" user="fx99">
                <member type="way" ref="28823678" role="outer"/>
                <member type="way" ref="30033834" role="outer"/>
                <member type="way" ref="195085889" role="outer"/>
                <member type="way" ref="196473438" role="outer"/>
                <member type="way" ref="194705249" role="outer"/>
                <member type="way" ref="195085890" role="outer"/>
                <member type="way" ref="69685291" role="outer"/>
                <member type="way" ref="69685244" role="outer"/>
                <member type="way" ref="30101361" role="outer"/>
                <member type="way" ref="196473436" role="outer"/>
                <member type="way" ref="45048519" role="outer"/>
                <member type="way" ref="194705245" role="outer"/>
                <member type="way" ref="196167289" role="outer"/>
                <member type="way" ref="196473435" role="outer"/>
                <member type="way" ref="42941597" role="outer"/>
                <member type="way" ref="30033065" role="outer"/>
                <member type="way" ref="30033066" role="outer"/>
                <member type="way" ref="196328310" role="outer"/>
                <member type="way" ref="196328308" role="outer"/>
                <member type="relation" ref="66998" role=""/>
                <member type="relation" ref="66999" role=""/>
                <member type="relation" ref="67000" role=""/>
                <tag k="admin_level" v="9"/>
                <tag k="boundary" v="administrative"/>
                <tag k="is_in" v="München,Bayern,Bundesrepublik Deutschland,Europe"/>
                <tag k="name" v="Stadtbezirk 04 Schwabing-West"/>
                <tag k="note" v="München, Stadtbezirk 4, 3 Bezirksteile"/>
                <tag k="type" v="multipolygon"/>
        </relation>

The file can be found on:
http://download.geofabrik.de/europe/germany/bayern.html

I use the following command:
./osm2pgsql/osm2pgsql --cache 5000 --number-processes 7 -S osm2pgsql/default.style --latlong -H localhost -U michael -P 9201 -d osm oberbayern-latest.osm.bz2

My version of osm2pgsql:
./osm2pgsql/osm2pgsql --version
osm2pgsql SVN version 0.83.0 (64bit id space)

I tried with native ubuntu 12.04 libraries, but now i got the latest ones as you can see at the end of my post (ldd command).

I tried with postgis 1.5 and 2.0.

I use postgres 8.4.

I configured osm2pgsql with 32 bit space and 64 bit space. No difference. Before I switchted to this version of osm2pgsql i could import everything, but had other problems (with appending spain to germany database). I don't know this old version any more now.

here are the used libraries

ldd osm2pgsql
        linux-vdso.so.1 =>  (0x00007fff7b1ff000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fce528bb000)
        libpq.so.5 => /usr/lib/libpq.so.5 (0x00007fce5268f000)
        libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007fce52333000)
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fce52123000)
        libgeos-3.4.0dev.so => /usr/local/lib/libgeos-3.4.0dev.so (0x00007fce51d89000)
        libproj.so.0 => /usr/lib/libproj.so.0 (0x00007fce51b46000)
        libprotobuf-c.so.0 => /usr/lib/libprotobuf-c.so.0 (0x00007fce51936000)
        liblua5.2.so.0 => /usr/lib/x86_64-linux-gnu/liblua5.2.so.0 (0x00007fce51705000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fce51404000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fce51108000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fce50ef2000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fce50cd4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fce50915000)
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fce506b7000)
        libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fce502db000)
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fce5000d000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fce4fe09000)
        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fce4fbca000)
        libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007fce4f97b000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fce4f777000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fce52ae3000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fce4f54e000)
        libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fce4f346000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fce4f141000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fce4ef25000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007fce4ed17000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007fce4eafb000)
        libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007fce4e8bd000)
        libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26 (0x00007fce4e601000)
        libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007fce4e382000)
        libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007fce4e17b000)
        libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007fce4def4000)
        libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007fce4dc54000)
        libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007fce4da20000)
        libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007fce4d80a000)
        libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3 (0x00007fce4d5f9000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fce4d3e7000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007fce4d1e2000)
        libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007fce4cfb9000)
        libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007fce4cdaa000)
        libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007fce4cb5f000)
        libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007fce4c8bc000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fce4c683000)

please help, i am really stuck here.
Thanks, Michael

Build fails with following exception:

https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-44

autoreconf-2.68: Entering directory `.'
autoreconf-2.68: configure.ac: not using Gettext
autoreconf-2.68: running: aclocal --force -I m4
autoreconf-2.68: configure.ac: tracing
autoreconf-2.68: running: libtoolize --copy --force
autoreconf-2.68: running: /usr/bin/autoconf-2.68 --force
autoreconf-2.68: running: /usr/bin/autoheader-2.68 --force
autoreconf-2.68: running: automake --add-missing --copy --force-missing
configure.ac:41: installing './config.guess'
configure.ac:41: installing './config.sub'
configure.ac:10: installing './install-sh'
configure.ac:10: installing './missing'
Makefile.am: installing './depcomp'
autoreconf-2.68: Leaving directory `.'
getconf: Unrecognized variable `LFS_CFLAGS'
configure: error: cannot find suitable Lua interpreter

Introduced by the following change:

3555d10

The integration tests fail with

SELECT AddGeometryColumn('planet_osm_point', 'way', 900913, 'POINT', 2 );
 failed: ERROR:  AddGeometryColumn() - invalid SRID
CONTEXT:  SQL statement "SELECT AddGeometryColumn('','',$1,$2,$3,$4,$5, $6)"
PL/pgSQL function "addgeometrycolumn" line 5 at SQL statement

because they do not add the 900913 SRID. Before doing an osm2pgsql import on a new database you have to set up postgis, hstore and the 900913 SRID

It's been said that you could almost use /dev/null for a flat-nodes file if you have a large enough -C value and don't plan to update the database. An option that allows you to use no file at all for flat nodes would be useful for when someone doesn't plan to update diffs and has enough memory. It would save on sequential writes and save on disk space. The latter is useful when using SSDs.

Hello,

Im using Postgres9.0.10 and PostGIS 1.5.5-1 witn CentOS5.5 32bits

I have installed these packages: geos-devel proj-devel postgresql-devel libxml2-devel bzip2-devel gcc-c++ autoconf automake libtool
But i had some problems with this: protobuf-c-devel (because im using centos 5.5)

./autogen.sh: seems to be ok

(at the end)
legacy/Makefile.am: installing ./depcomp' autoreconf: Leaving directory.'

./configure:
(of course)
checking for protobuf-c 0.14...
checking for protobuf-c version >= 0.14...
checking for ProtobufCFieldDescriptor.packed... no
protobuf-c >= 0.14: no
checking for protobuf-c usability... no
(and)
checking for PostgreSQL libraries... yes
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... no
checking whether pthreads work with -Kthread... no
checking whether pthreads work with -kthread... no
checking for the pthreads library -llthread... no
checking whether pthreads work with -pthread... yes
(to the end)
config.status: creating Makefile
config.status: creating legacy/Makefile
config.status: creating config.h
config.status: executing depfiles commands

make:
(a lot of references error of pgsql8.4 sources...)
(to the end)
/usr/lib/libpq.a(fe-auth.o): In function pg_krb5_sendauth': /var/lib/pgsql/rpm/BUILD/postgresql-8.4.14/src/interfaces/libpq/fe-auth.c:220: undefined reference tokrb5_sname_to_principal'
/var/lib/pgsql/rpm/BUILD/postgresql-8.4.14/src/interfaces/libpq/fe-auth.c:225: undefined reference to error_message' /var/lib/pgsql/rpm/BUILD/postgresql-8.4.14/src/interfaces/libpq/fe-auth.c:243: undefined reference tokrb5_free_principal'
/var/lib/pgsql/rpm/BUILD/postgresql-8.4.14/src/interfaces/libpq/fe-auth.c:248: undefined reference to krb5_sendauth' /var/lib/pgsql/rpm/BUILD/postgresql-8.4.14/src/interfaces/libpq/fe-auth.c:278: undefined reference tokrb5_free_error'
/var/lib/pgsql/rpm/BUILD/postgresql-8.4.14/src/interfaces/libpq/fe-auth.c:283: undefined reference to krb5_free_principal' /var/lib/pgsql/rpm/BUILD/postgresql-8.4.14/src/interfaces/libpq/fe-auth.c:273: undefined reference toerror_message'
collect2: ld returned 1 exit status
make[2]: *** [osm2pgsql] Error 1
make[2]: Leaving directory /home/covoit/workspace/osm2pgsql/osm2pgsql' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory/home/covoit/workspace/osm2pgsql/osm2pgsql'
make: *** [all] Error 2

Currently osm2pgsql always creates geometry indexes and orders the tables by geometry, essentially clustering them. There are cases where this is not desirable.

1. Benchmarking of other parts.
2. Analysis and tasks with non-standard queries. You might want geometry indicies but might prefer to cluster by something else, or you might want to query something like the average area of a building poly in France, which would involve a sequential scan and not benefit from geom indices.
3. Consuming a substantial set of diffs. I believe diff processing is faster without geom indices, and the ordering is more effective if done after updating.
4. Low space situations where you don't have room to order the tables.

5. Cases where you don't plan to consume diffs (e.g. imports with --drop). You can be better off with a non-default FILLFACTOR

https://github.com/openstreetmap/osm2pgsql/blob/master/output-pgsql.c#L160 implies that wildcards (* or ?) can be used in delete style entries. However I can't actually find any code that handles wildcards in delete.

Do we accept wildcards in deletes? If not, should we? It would be nice to be able to drop source* rather than source, source:ja, source:en, source:name, ... etc

See #32 (comment)

An increasing number of users are using hstore. Most of them won't want tags like the TIGER tags in their hstore tag columns. I suggest having the default.style not import those to the hstore by default. Note: errol's DB is one of the exceptions where we probably want every tag.

@Komzpa has a list of common tags that are unlikely to ever be used for rendering

You should be able to cut out a bunch of the if statements this way.

Add before the function definitions:

polygon_keys = { 'building', 'landuse', 'amenity', 'harbour', 'historic', 'leisure', 
                 'man_made', 'military', 'natural', 'office', 'place', 'power',
                 'public_transport', 'shop', 'sport', 'tourism', 'waterway',
                 'wetland', 'water', 'aeroway' }

then replace the if statements in filter_tags_way with:

    for i,k in ipairs(polygon_keys) do
        if keyvalues[k] then
            poly=1
            break
        end
    end

The code was moved in d32bad6 but the comment remains:

https://github.com/openstreetmap/osm2pgsql/blob/6f7253ebace4197c8307dcc6c388a618d9b892d5/output-pgsql.c#L88

While following the directions for installing osm2pgsql from source, I get a fatal error while running 'make':

build_geometry.Tpo -c -o build_geometry.o build_geometry.cpp
build_geometry.cpp:29:26: fatal error: geos/version.h: No such file or directory
compilation terminated.

I'm following the directions here:
http://wiki.openstreetmap.org/wiki/Osm2pgsql

and here:
http://wiki.openstreetmap.org/wiki/Osm2pgsql#From_source_.28generic.29

I'm building libprotobuf-c0-dev from source as well, because I'd like PBF support.

Thanks.

I don't think it belongs to this utility granting permissions on tables, instead output_pgsql.c is granting SELECT to PUBLIC (any database user) on anything that's imported.

On CartoDB we have a special user representing "the public", and that user is NOT "PUBLIC", so this assumption of osm2pgsql is wrong. What's the rationale for setting permissions from the importer ?

Would it be possible to select feature to import that do not match a column in the tables? Or is it already possible with lua style? I think this is would be very useful when having all tags in an indexed hstore.
I could look into implementing this with some pointers how it should be done

The parsing of tags for multipolygons seems broken - I'm getting man_made=pier on polygon created from relation 936128 (admin boundary for Poland). This is a very large multipolygon - out of several hundreds of way members only two (way 227911442 and 227911443) have this tag.

I'm using osm2pgsql build from git on 2013-09-16.

Invalidation of tiles for large relations could lead to all the tiles in the US being invalidated by a tag change on a US border relation. It would be desirable to be able to not invalidate the interior of large boundary relations.

cc @gravitystorm

fastupdate does not work with osm2pgsql because with a high work_mem it builds up a large temporary part of the index it has to sequentially scan, slowing down updates significantly

It would be nice if we could use fastupdate with a smaller temporary part of the index.

Difference found: ways in a MP where inner, outer and relation are tagged with landuse=forest. I don't have a problem with lua's handling, the tagging is unclear. Also, a broken import.

Way in question: http://www.openstreetmap.org/browse/way/42234367

I'm hitting an issue with some tags that are not correctly imported in table planet_osm_polygons, and I think it might be because of the tags order in the .osm source.

For example, these two osm files only differ on the tag ordering of type=boundary:
http://osm7.openstreetmap.fr/~osm2pgsql/alsace.osm.gz
http://osm7.openstreetmap.fr/~osm2pgsql/alsace-fail.osm.gz

--- alsace.osm  2013-06-01 00:38:25.182404624 +0200
+++ alsace-fail.osm     2013-06-01 01:29:07.841660538 +0200
@@ -31910,6 +31910,7 @@
     <member type="way" ref="48683073" role="outer"/>
     <member type="way" ref="48565263" role="outer"/>
     <member type="way" ref="23301554" role="outer"/>
+    <tag k="type" v="boundary"/>
     <tag k="admin_level" v="4"/>
     <tag k="alt_name:la" v="Elisatia"/>
     <tag k="boundary" v="administrative"/>
@@ -31933,7 +31934,6 @@
     <tag k="ref:INSEE" v="42"/>
     <tag k="ref:ISO 3166-2" v="A"/>
     <tag k="ref:NUTS" v="FR42"/>
-    <tag k="type" v="boundary"/>
     <tag k="wikipedia" v="fr:Alsace"/>
   </relation>

With the second file, admin_level column of planet_osm_polygon is not filled with value 4.

Must the tags be sorted before importing with osm2pgsql ?

I am merging two planets from planet.openstreetmap.nl. The planets for Aruba and Curacao. Although both islands are pretty far apart, they share one common relations; the nautical boundary.

During import in regular mode, all goes well. But in slim mode, I am getting errors:

--2013-04-08 15:43:51-- http://planet.openstreetmap.nl/aruba/planet-aruba-130408.osm.pbf
Resolving planet.openstreetmap.nl (planet.openstreetmap.nl)... 2a00:d10:101::13:1, 93.186.179.161
Connecting to planet.openstreetmap.nl (planet.openstreetmap.nl)|2a00:d10:101::13:1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1069130 (1.0M)
Saving to: ‘planet-aruba-130408.osm.pbf’

100%[===============================================================================================================================================================================>] 1,069,130 1.83MB/s in 0.6s

2013-04-08 15:43:52 (1.83 MB/s) - ‘planet-aruba-130408.osm.pbf’ saved [1069130/1069130]

osm2pgsql SVN version 0.81.0 (64bit id space)

Using projection SRS 4326 (Latlong)
Setting up table: planet_osm_point
NOTICE: table "planet_osm_point_tmp" does not exist, skipping
Setting up table: planet_osm_line
NOTICE: table "planet_osm_line_tmp" does not exist, skipping
Setting up table: planet_osm_polygon
NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping
Setting up table: planet_osm_roads
NOTICE: table "planet_osm_roads_tmp" does not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=800MB, maxblocks=102401*8192, allocation method=11
Mid: pgsql, scale=10000000 cache=800
Setting up table: planet_osm_nodes
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_nodes_pkey" for table "planet_osm_nodes"
Setting up table: planet_osm_ways
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_ways_pkey" for table "planet_osm_ways"
Setting up table: planet_osm_rels
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_rels_pkey" for table "planet_osm_rels"

Reading in file: planet-aruba-130408.osm.pbf
Processing: Node(112k 112.2k/s) Way(8k 4.40k/s) Relation(50 50.00/s) parse time: 3s

Node stats: total(112174), max(2170360891) in 1s
Way stats: total(8792), max(196359215) in 2s
Relation stats: total(51), max(2323309) in 0s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads

Going over pending ways...
4885 ways are pending

Using 1 helper-processes
Helper process 0 out of 1 initialised
Process 0 finished processing 4885 ways in 4 sec

All child processes exited

4885 Pending ways took 4s at a rate of 1221.25/s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads

Going over pending relations...
0 relations are pending

Using 1 helper-processes
Process 0 finished processing 0 relations in 0 sec

All child processes exited

node cache: stored: 112174(100.00%), storage efficiency: 50.92% (dense blocks: 18, sparse nodes: 100935), hit rate: 100.00%
Sorting data and creating indexes for planet_osm_roads
Sorting data and creating indexes for planet_osm_point
Sorting data and creating indexes for planet_osm_polygon
Sorting data and creating indexes for planet_osm_line
Stopping table: planet_osm_nodes
Stopping table: planet_osm_ways
Building index on table: planet_osm_ways (fastupdate=off)
Stopping table: planet_osm_rels
Building index on table: planet_osm_rels (fastupdate=off)
Stopped table: planet_osm_nodes in 0s
Stopped table: planet_osm_rels in 0s
Analyzing planet_osm_roads finished
Analyzing planet_osm_line finished
Analyzing planet_osm_polygon finished
Analyzing planet_osm_point finished
Stopped table: planet_osm_ways in 0s
Copying planet_osm_roads to cluster by geometry finished
Creating geometry index on planet_osm_roads
Creating osm_id index on planet_osm_roads
Copying planet_osm_line to cluster by geometry finished
Creating geometry index on planet_osm_line
Creating indexes on planet_osm_roads finished
Creating osm_id index on planet_osm_line
All indexes on planet_osm_roads created in 2s
Completed planet_osm_roads
Copying planet_osm_polygon to cluster by geometry finished
Creating geometry index on planet_osm_polygon
Creating osm_id index on planet_osm_polygon
Creating indexes on planet_osm_line finished
All indexes on planet_osm_line created in 2s
Completed planet_osm_line
Creating indexes on planet_osm_polygon finished
Copying planet_osm_point to cluster by geometry finished
Creating geometry index on planet_osm_point
All indexes on planet_osm_polygon created in 2s
Completed planet_osm_polygon
Creating osm_id index on planet_osm_point
Creating indexes on planet_osm_point finished
All indexes on planet_osm_point created in 4s
Completed planet_osm_point

Osm2pgsql took 15s overall

Reading in file: planet-curacao-130408.osm.pbf
Processing: Node(34k 34.7k/s) Way(5k 5.02k/s) Relation(30 30.00/s) parse time: 1s

Node stats: total(34735), max(2229592357) in 1s
Way stats: total(5020), max(213219417) in 0s
Relation stats: total(33), max(2676830) in 0s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
COPY_END for COPY planet_osm_rels FROM STDIN;
failed: ERROR: duplicate key value violates unique constraint "planet_osm_rels_pkey"
DETAIL: Key (id)=(2323309) already exists.
CONTEXT: COPY planet_osm_rels, line 30: "2323309 1 354 {268396336,202476411,202476412,86298925,202487355,202487361,202487362,86297905,8629810..."

Currently we have a nocache flag. This flag sets FLAG_NOCACHE for the tag, but the only other occurrence of FLAG_NOCACHE is where it is defined to be 4, i.e. the flag is not actually used.

This is confusing, and probably would lead to errors if anyone actually tried to do anything with it.

The docs say

In the future I would like to be able to add directives like "nocache" which tells osm2pgsql that it is unlikely this node will be used by a way and so it doesn't need to be stored (eg coastline nodes). While in essence an optimisation hack, for --slim mode it doesn't matter if you're wrong, but in non-slim you might break something!

I suggest removing nocache from the default.style, to eventually allow the option of the removal of the FLAG_NOCACHE code.

Copying planet_osm_polygon to cluster by geometry finished
CREATE INDEX planet_osm_polygon_no_building_index ON planet_osm_polygon USING GIST (way) WHERE building is null TABLESPACE gisidx;
failed: ERROR: syntax error at or near "TABLESPACE"
LINE 1: ...m_polygon USING GIST (way) WHERE building is null TABLESPACE...
^

The TABLESPACE spec belongs before the WHERE clause.

The logic for what goes in the planet_osm_roads table is highly tied to the default OSM Mapnik stylesheet. For rendering other styles it might make sense for this table to be different or not present at all.

Completely removing the creation of this table (since I don't need it) cuts 10-20% off my import times.

This may not be a bug, but wiki here
http://wiki.openstreetmap.org/wiki/Osm2pgsql
says way_area is "calculated in Mercator meters".

Maybe this is a feature and way_area should really be calculated in decimal degrees when importing with -l flag, but then it shoud be pointed out, because some major osm styles use this field when filtering out data for rendering. Or could it be another flag for convenience?:)

node_persistent_cache.c tests for a non-zero return value from posix_fallocate but then relies on errno being set; posics_fallocate however does not set errno; instead the return value must be interpreted for determining the cause of error.

#32 mentions phstore as a flag for .style files, but I'm sure there are others undocumented. These should be.

Hello,
is it normal behaviour, that the same osm_id is used multiple times in the database? I wanted to use the osm_id as primary key, but this makes it problematic now.

osm=# SELECT * from planet_osm_polygon where name = 'München';
 osm_id | note | source | created_by | access | addr:housename | addr:housenumber | addr:interpolation | admin_level | aerialway | aeroway | amenity | area | barrier | bicycle | brand | bridge |    boundary    | building | construction | covered | culvert | cutting | denomination | disused | embankment | foot | generator:source | harbour | highway | historic | horse | intermittent | junction | landuse | layer | leisure | lock | man_made | military | motorcar |  name   | natural | office | oneway | operator | place | population | power | power_source | public_transport | railway | ref | religion | route | service | shop | sport | surface | toll | tourism | tower:type | tracktype | tunnel | water | waterway | wetland | width | wood | z_order | way_area | timezone |                                                                                                                                                                                way                                                                                                                                                                                 
--------+------+--------+------------+--------+----------------+------------------+--------------------+-------------+-----------+---------+---------+------+---------+---------+-------+--------+----------------+----------+--------------+---------+---------+---------+--------------+---------+------------+------+------------------+---------+---------+----------+-------+--------------+----------+---------+-------+---------+------+----------+----------+----------+---------+---------+--------+--------+----------+-------+------------+-------+--------------+------------------+---------+-----+----------+-------+---------+------+-------+---------+------+---------+------------+-----------+--------+-------+----------+---------+-------+------+---------+----------+----------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 -62428 |      |        |            |        |                |                  |                    | 6           |           |         |         |      |         |         |       |        | administrative |          |              |         |         |         |              |         |            |      |                  |         |         |          |       |              |          |         |       |         |      |          |          |          | München |         |        |        |          |       |            |       |              |                  |         |     |          |       |         |      |       |         |      |         |            |           |        |       |          |         |       |      |       0 |        0 |          | 0103000020E6100000010000000A00000087E8C6AAF7FA2640FC766DCA700948408D367D2C33FB264092CF2B9E7A0948408B86318E36FB26406231EA5A7B094840DE23F66459FB2640D56E055E770948405473B9C150FB2640357D76C075094840E72ED3403DFB26401F8FCF1A72094840F28B5C8132FB2640BBCF961870094840E784758824FB2640BA2583ED71094840744694F606FB2640E6FE8FB86C09484087E8C6AAF7FA2640FC766DCA70094840
 -62428 |      |        |            |        |                |                  |                    | 6           |           |         |         |      |         |         |       |        | administrative |          |              |         |         |         |              |         |            |      |                  |         |         |          |       |              |          |         |       |         |      |          |          |          | München |         |        |        |          |       |            |       |              |                  |         |     |          |       |         |      |       |         |      |         |            |           |        |       |          |         |       |      |       0 |    1e-06 |          | 0103000020E61000000100000006000000679EB70C931C274089FB7E202F0A48404B2B7414D61C27405ECA0A8F470A4840E79CE96F531D2740B62BF4C1320A48400675DBCF731D2740CB1E57D92D0A48404DF8A57EDE1C2740E1AD98B6240A4840679EB70C931C274089FB7E202F0A4840
(2 rows)

configure line 15698 should be

        as_fn_error $? "$ac_word does not exist or it is not an executable file" "$LINENO" 5

or xml2-config name is not printed in error msg.

package lua5.2 is not suggested nor by configure and neither in readme.

in readme is not suggested to do make install

900913 is included in PostGIS 2.0.1:
duplicate key value violates unique constraint "spatial_ref_sys_pkey"
DETAIL: Key (srid)=(900913) already exists.

a link to
http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
could be useful. checkpoint_completion_target is also interesting, even if I'm not sure I've understood what it does.

It's not very clear of the purpose of not having --melts-geometry , is it a problem with old versions of pg / postgis? Furthermore on this wiki page:
http://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks
it's stated that pbf reader is 2x and (bugged) primitive is 30% faster. IMHO it's useful to document both of them in readme and man.

PS: thank you very much for your program and work :)

No import any nodes where id>type integer in table planet_osm_nodes.id and planet_osm_ways.nodes. After changed type INTEGER -> NUMERIC show error:

PREPARE node_changed_mark(int4) AS UPDATE planet_osm_ways SET pending = true WHERE nodes && ARRAY[$1] AND NOT pending;
failed: ERROR: operator does not exist: numeric[] && integer[]
LINE 1: ...TE planet_osm_ways SET pending = true WHERE nodes && ARRAY[$...
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.

The README included in the repository needs updating and improving.

Newer feature like e.g. lua tag transform need explaining and also the build dependencies updated.

Give example of typically used parameter combinations. The default settings usually aren't appropriate for full planet imports and can lead to very poor performance. So documenting typical parameter combinations is important.