Code Monkey home page Code Monkey logo

gosmparse's Introduction

OpenStreetMap PBF Parser in Go

GoDoc

gosmparse uses a callback driven API, which is stable (Documentation).

It has been designed with performance and maximum usage convenience in mind; on an Intel Core i7-6820HQ with NVMe flash it is able to process ~75 MB/s, so a planet file can be processed in under 10 minutes. If you find possible speed-ups or other improvements, let me know.

Characteristics

  • fast
  • tested with different files from different sources/generators
  • more than 85% test coverage and benchmarks for all hot spots
  • one dependency only: protobuf package (a few more are used by tests and are included in the module)
  • can read from any io.Reader (e.g. for parsing during download)
  • supports history files

Non-Features

  • Does not build geometries
  • No element cache

Install

go get -u github.com/thomersch/gosmparse

Example Usage

// Implement the gosmparser.OSMReader interface here.
// Streaming data will call those functions.
type dataHandler struct{}

func (d *dataHandler) ReadNode(n gosmparse.Node)         {}
func (d *dataHandler) ReadWay(w gosmparse.Way)           {}
func (d *dataHandler) ReadRelation(r gosmparse.Relation) {}

func ExampleNewDecoder() {
	r, err := os.Open("filename.pbf")
	if err != nil {
		panic(err)
	}
	dec := gosmparse.NewDecoder(r)
	// Parse will block until it is done or an error occurs.
	err = dec.Parse(&dataHandler{})
	if err != nil {
		panic(err)
	}
}

Download & Parse

It is possible to parse during download, so you don't have to wait for a download to finish to be able to start the parsing/processing. You can simply use the standard Go net/http package and pass resp.Body to the decoder.

resp, err := http.Get("http://download.geofabrik.de/europe/germany/bremen-latest.osm.pbf")
if err != nil {
	panic(err)
}
defer resp.Body.Close()
dec := gosmparse.NewDecoder(resp.Body)
err = dec.Parse(&dataHandler{})
if err != nil {
	panic(err)
}

Did it break?

If you found a case, where gosmparse broke, please report it and provide the file that caused the failure.

gosmparse's People

Contributors

aleung avatar brendanashworth avatar thomersch avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gosmparse's Issues

No context?

Excellent work - but there doesn't seem to be any context.Context. Won't Goroutines leak on aborts?

Is there any particular reason that contexts aren't used? Performance?

Ensuring all nodes are handled before ways

Heya,

Is there any mechanism that would allow me to ensure that all calls to ReadNode have been completed before the first time ReadWay is called?

I would like to denormalize ways, so I need to ensure that all the nodes are in memory before processing the ways.

Cancellable `Decoder.Parse`

Perhaps I am missing some functionality, but wouldn't it be helpful if Parse took a context so that it can be cancelled? Currently the only way to stop the parsing is to panic. It would be good to be able to gracefully terminate the parsing as it is a long running task.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.