mapquest / osm2pgsql Goto Github PK

View Code? Open in Web Editor NEW

This project forked from osm2pgsql-dev/osm2pgsql

2.0 12.0 0.0 32.44 MB

OpenStreetMap data to PostgreSQL converter

Home Page: http://wiki.openstreetmap.org/wiki/Osm2pgsql

License: GNU General Public License v2.0

Shell 0.39% C++ 94.96% Lua 1.20% C 0.12% Python 2.67% Protocol Buffer 0.65%

osm2pgsql's Introduction

osm2pgsql

osm2pgsql is a tool for loading OpenStreetMap data into a PostgreSQL / PostGIS database suitable for applications like rendering into a map, geocoding with Nominatim, or general analysis.

Features

Converts OSM files to a PostgreSQL DB
Conversion of tags to columns is configurable in the style file
Able to read .gz, .bz2, .pbf and .o5m files directly
Can apply diffs to keep the database up to date
Support the choice of output projection
Configurable table names
Gazetteer back-end for Nominatim
Support for hstore field type to store the complete set of tags in one database field if desired

Installing

The latest source code is available in the OSM git repository on github and can be downloaded as follows:

$ git clone git://github.com/openstreetmap/osm2pgsql.git

Building

Osm2pgsql uses the GNU Build System to configure and build itself and requires

expat
geos
proj
bzip2
zlib
Protocol Buffers
PostgreSQL client libraries
Lua (Optional, used for Lua tag transforms)

It also requires access to a database server running PostgreSQL and PostGIS.

Make sure you have installed the development packages for the libraries mentioned in the requirements section and a C++ compiler which supports C++11. Both GCC 4.8 and Clang 3.4 meet this requirement.

To install on a Debian or Ubuntu system, first install the prerequisites:

sudo apt-get install autoconf automake libtool make g++ pkg-config libboost-dev \
  libboost-system-dev libboost-filesystem-dev libboost-thread-dev libexpat1-dev \
  libgeos-dev libgeos++-dev libpq-dev libbz2-dev libproj-dev zlib1g-dev \
  protobuf-compiler libprotobuf-dev lua5.2 liblua5.2-dev

To install on a Fedora system, use

sudo yum install gcc-c++ automake libtool pkgconfig boost-devel \
  expat-devel bzip2-devel postgresql-devel geos-devel proj-devel \
  lua-devel protobuf-devel protobuf-lite-devel

To install on a FreeBSD system, use

pkg install devel/git devel/autoconf devel/automake devel/gmake devel/libtool \
  textproc/expat2 graphics/geos graphics/proj databases/postgresql94-client \
  devel/boost-libs devel/protobuf lang/lua52 devel/pkgconf

Then you should be able to bootstrap the build system:

./autogen.sh

And then run the standard GNU build install:

./configure && make && make install

Please see ./configure --help for more options on how to control the build process.

On FreeBSD instead bootstrap and then run

LUA_LIB=`pkg-config --libs lua-5.2` ./configure && gmake && gmake install

Usage

Osm2pgsql has one program, the executable itself, which has 42 command line options.

Before loading into a database, the database must be created and the PostGIS and optionally hstore extensions must be loaded. A full guide to PostgreSQL setup is beyond the scope of this readme, but with reasonably recent versions of PostgreSQL and PostGIS this can be done with

createdb gis
psql -d gis -c 'CREATE EXTENSION postgis; CREATE EXTENSION hstore;'

A basic invocation to load the data into the database gis for rendering would be

osm2pgsql --create --database gis data.osm.pbf

This will load the data from data.osm.pbf into the planet_osm_point, planet_osm_line, planet_osm_roads, and planet_osm_polygon tables.

When importing a large amount of data such as the complete planet, a typical command line would be

osm2pgsql -c -d gis --slim -C <cache size> \
  --flat-nodes <flat nodes> planet-latest.osm.pbf

where

<cache size> is 24000 on machines with 32GiB or more RAM or about 75% of memory in MiB on machines with less
<flat nodes> is a location where a 24GiB file can be saved.

The databases from either of these commands can be used immediately by Mapnik for rendering maps with standard tools like renderd/mod_tile, TileMill, Nik4, among others. It can also be used for spatial analysis or shapefile exports.

Additional documentation is available on writing command lines.

Alternate backends

In addition to the standard pgsql backend designed for rendering there is also the gazetteer database for geocoding, principally with Nominatim, and the null backend for testing. For flexibility a new multi backend is also avialable which allows the configuration of custom postgres tables instead of those provided in the pgsql backend.

Any questions should be directed at the osm dev list http://wiki.openstreetmap.org/index.php/Mailing_lists

Contributing

We welcome contributions to osm2pgsql. If you would like to report an issue, please use the issue tracker on GitHub.

More information can be found in CONTRIBUTING.md.

General queries can be sent to the tile-serving@ or dev@ mailing lists.

osm2pgsql's People

Contributors

Stargazers

Watchers

osm2pgsql's Issues

Dependency on default.style being installed

This seems to be specific to our multi/C++ branch.

If you run make test on a machine which doesn't have osm2psql installed, you get the error

Couldn't open style file '/usr/local/share/osm2pgsql/default.style': No such file or directory

Repeated output when running in screen detached

When running in screen detached when I come back during the processing stage I get a screen full of text

 Relation(1975430 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975440 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975450 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975460 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975470 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975480 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975490 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975500 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975510 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975520 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975530 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975540 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975550 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975560 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975570 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975580 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975590 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975600 220.Processing: Node(2013054k 2207.3k/s) Way(196145k 39.62k/s) Relation(1975610 220

If I hit enter, it continues working normally) Note: I was getting ~220/s relations, which is probably the cause of the 220

Storage efficiency reporting

C++ branch

node cache: stored: 0(-nan%), storage efficiency: -nan% (dense blocks: 0, sparse nodes: 0), hit rate: 0.00%
Maximum node in persistent node cache: 2445177855
node cache: stored: 0(-nan%), storage efficiency: -nan% (dense blocks: 0, sparse nodes: 0), hit rate: 0.00%
Maximum node in persistent node cache: 2445176831
node cache: stored: 0(-nan%), storage efficiency: -nan% (dense blocks: 0, sparse nodes: 0), hit rate: 0.00%
Maximum node in persistent node cache: 2445177855
node cache: stored: 0(-nan%), storage efficiency: -nan% (dense blocks: 0, sparse nodes: 0), hit rate: 0.00%
Maximum node in persistent node cache: 2445177855
node cache: stored: 0(-nan%), storage efficiency: -nan% (dense blocks: 0, sparse nodes: 0), hit rate: 0.00%
Maximum node in persistent node cache: 2445177855
node cache: stored: 0(-nan%), storage efficiency: -nan% (dense blocks: 0, sparse nodes: 0), hit rate: 0.00%
Maximum node in persistent node cache: 2445177855
node cache: stored: 0(-nan%), storage efficiency: -nan% (dense blocks: 0, sparse nodes: 0), hit rate: 0.00%
Maximum node in persistent node cache: 2445177855
node cache: stored: 0(-nan%), storage efficiency: -nan% (dense blocks: 0, sparse nodes: 0), hit rate: 0.00%

We should only report it once, and given how long it took, this particular run had a 100% cache hit rate.

Sequential scans of ways_pending table

During the processing ways portion of the import with the multi-backend branch Postgres is stuck doing sequential scans of the ways_pending table, bringing the ways rate down to 10s/second or slower.

PREPARE set_mark(int8) AS 
  INSERT INTO "planet_osm_ways_pending" (id) 
    SELECT $1 WHERE NOT EXISTS 
      (SELECT id FROM "planet_osm_ways_pending" WHERE id = $1)

(Ignore the fact that it's a PREPARE)

\d+ planet_osm_ways_pending shows that this table has no indexes, explaining the sequential scans.

I tried manually building an index for debugging, but it's in transaction and got a lock, so I couldn't.

It's not 100% obvious to me that we need the NOT EXISTS check as an OSM file will only ever have one ID for an object type, and if the file is invalid, building a UNIQUE index after loading data will catch it. On the other hand, there might be an issue with importing multiple files that have some overlap.

Missing tablespace text

The test needs a temporary tablespace to run in, but it does not exist. Please create the temporary tablespace. On Linux, you can do this by running:
  sudo mkdir -p /tmp/psql-tablespace
  sudo /bin/chown postgres.postgres tmp/psql-tablespace
  psql -c "CREATE TABLESPACE tablespacetest LOCATION '/tmp/psql-tablespace'" postgres

Missing a slash on the second occurrence of tmp/psql-tablespace.

Missing libboost-thread-dev in dependencies

We need libboost-thread-dev but it's not listed in the readme.

MPs wrongly imported with minimal style file

This is part of the cause of a make check failure in C++/multi

I ran
./osm2pgsql -S buildings.style ~/osm/liechtenstein-latest.osm.pbf where buildings.style is node,way building text polygon.

Running the SQL select count(*) from planet_osm_polygon where building is null; gives me the result 41, which is clearly absurd.

Edit: Fixed commands

Start middle table indexing once processing is finished

With in-memory pending, the middle tables are only read from once processing is finished. This allows us to build indexes right away, including the GIN index on planet_osm_ways.nodes, which is often slower than the rest of the import.

By starting it sooner, it will make the pending ways and other indexes no longer part of the critical path for import speed on a --slim import. It might increase contention for disk, but this is probably not an issue.

Not planned for resolution pre-merge

Compilation error FreeBSD clang 3.3

libtool: compile:  c++ -DHAVE_CONFIG_H -I. -D_THREAD_SAFE -pthread -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/include -I/usr/include -I/usr/local/include -I/usr/local/include -DOSM2PGSQL_DATADIR=\"/usr/local/share/osm2pgsql\" -Igeos-fallback -pthread -I/usr/local/include -g -O2 -MT osmdata.lo -MD -MP -MF .deps/osmdata.Tpo -c osmdata.cpp  -fPIC -DPIC -o .libs/osmdata.o
In file included from osmdata.cpp:1:
In file included from ./osmdata.hpp:9:
In file included from ./output.hpp:25:
/usr/local/include/boost/lockfree/queue.hpp:91:5: error: implicit instantiation of undefined template 'boost::STATIC_ASSERTION_FAILURE<false>'
    BOOST_STATIC_ASSERT((boost::has_trivial_assign<T>::value));
    ^
/usr/local/include/boost/static_assert.hpp:169:13: note: expanded from macro 'BOOST_STATIC_ASSERT'
            sizeof(::boost::STATIC_ASSERTION_FAILURE< BOOST_STATIC_ASSERT_BOOL_CAST( __VA_ARGS__ ) >)>\
            ^
osmdata.cpp:353:21: note: in instantiation of template class 'boost::lockfree::queue<std::__1::pair<long, unsigned long>, boost::parameter::void_,
      boost::parameter::void_, boost::parameter::void_>' requested here
    pending_queue_t queue;
                    ^
/usr/local/include/boost/static_assert.hpp:87:26: note: template is declared here
template <bool x> struct STATIC_ASSERTION_FAILURE;
                         ^
osmdata.cpp:390:28: error: non-const lvalue reference to type 'middle_t::pending_processor' cannot bind to a value of unrelated type
      '<anonymous>::pending_threaded_processor'
        mid->iterate_ways( ptp );
                           ^~~
./middle.hpp:57:50: note: passing argument to parameter 'pf' here
    virtual void iterate_ways(pending_processor& pf) = 0;
                                                 ^
osmdata.cpp:395:33: error: non-const lvalue reference to type 'middle_t::pending_processor' cannot bind to a value of unrelated type
      '<anonymous>::pending_threaded_processor'
        mid->iterate_relations( ptp );
                                ^~~
./middle.hpp:58:55: note: passing argument to parameter 'pf' here
    virtual void iterate_relations(pending_processor& pf) = 0;

I'm just noting it down since I may not get to fixing it tonight