deloitteoptimalreality / lightosm.jl Goto Github PK

I've noticed that for some geographical regions there exists building data without relation data in the downloaded *.osm file.

For reference see https://www.openstreetmap.org/search?query=greenville%20ohio#map=18/40.10042/-84.62916

using LightOSM

download_osm_buildings(:place_name; 
   place_name = "greenville, ohio",    
   metadata = true,
   download_format = :osm,
   save_to_file_location = "buildings.osm")

buildings = buildings_from_file("buildings.osm");

ERROR: KeyError: key "relation" not found
Stacktrace:
 [1] getindex
   @ ./dict.jl:481 [inlined]
 [2] parse_osm_buildings_dict(osm_buildings_dict::Dict{String, Any})
   @ LightOSM ~/.julia/dev/LightOSM/src/buildings.jl:204
 [3] buildings_from_object
   @ ~/.julia/dev/LightOSM/src/buildings.jl:260 [inlined]
 [4] buildings_from_file(file_path::String)
   @ LightOSM ~/.julia/dev/LightOSM/src/buildings.jl:294

So I get that a way represents a set of edges between nodes in an OSM graph. I was wondering, is there a method to query all of the points that fall along a way? That is, I would like to identify all of the latitude and longitude points along a way.

I did google this question, and it seems like other people push the OSM graph into a database like PostGIS to do this. I was just wondering if there was a way to do this without pulling down to a database? Thanks.

There was an issue on line 418 of graph.jl: g.dijkstra_states = Vector{U}(undef, n) with an Int32 to Array{Int32, 1} implicit conversion not being possible. Was this supposed to be as follows?
g.dijkstra_states = Vector{Vector{U}}(undef, n)

This is an issue in setup of LightOSM.jl by following the tutorial jupyter notebook.

Steps I followed:

1. Created a python virtual environment in my project directory.
2. Pip installed pydeck using the forked version of deck.gl in the python virtual environment
3. Created a Julia environment at the project level and installed PyCall.jl
4. Set the PyCall build to the virtual environment python version(see screenshot)

After doing these, whenever it runs the pydeck part of building the layer and defining the tooltip I get the following.

PyError ($(Expr(:escape, :(ccall(#= /home/tcluri/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'NameError'>
NameError("name 'is_array' is not defined")
  File "/home/tcluri/projects/osmroute/.venv/lib/python3.8/site-packages/pydeck/bindings/layer.py", line 113, in __init__
    self.data = data
  File "/home/tcluri/projects/osmroute/.venv/lib/python3.8/site-packages/pydeck/bindings/layer.py", line 132, in data
    self._data = all_numpy_to_list(data_set)
  File "/home/tcluri/projects/osmroute/.venv/lib/python3.8/site-packages/pydeck/bindings/json_tools.py", line 94, in all_numpy_to_list
    if is_array(obj):


Stacktrace:
  [1] pyerr_check
    @ ~/.julia/packages/PyCall/7a7w0/src/exception.jl:62 [inlined]
  [2] pyerr_check
    @ ~/.julia/packages/PyCall/7a7w0/src/exception.jl:66 [inlined]
  [3] _handle_error(msg::String)
    @ PyCall ~/.julia/packages/PyCall/7a7w0/src/exception.jl:83
  [4] macro expansion
    @ ~/.julia/packages/PyCall/7a7w0/src/exception.jl:97 [inlined]
  [5] #107
    @ ~/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:43 [inlined]
  [6] disable_sigint
    @ ./c.jl:458 [inlined]
  [7] __pycall!
    @ ~/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:42 [inlined]
  [8] _pycall!(ret::PyObject, o::PyObject, args::Tuple{String, Vector{Dict{String, Any}}}, nargs::Int64, kw::PyObject)
    @ PyCall ~/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:29
  [9] _pycall!(ret::PyObject, o::PyObject, args::Tuple{String, Vector{Dict{String, Any}}}, kwargs::Base.Iterators.Pairs{Symbol, Any, NTuple{14, Symbol}, NamedTuple{(:pickable, :opacity, :stroked, :filled, :line_width_min_pixels, :line_width_max_pixels, :line_width_scale, :auto_highlight, :get_position, :get_radius, :radius_scale, :get_line_width, :get_line_color, :get_fill_color), Tuple{Bool, Float64, Bool, Bool, Int64, Int64, Int64, Bool, Vector{String}, Int64, Int64, Int64, Vector{Int64}, Vector{Int64}}}})
    @ PyCall ~/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:11
 [10] #_#114
    @ ~/.julia/packages/PyCall/7a7w0/src/pyfncall.jl:86 [inlined]
 [11] top-level scope
    @ In[25]:17
 [12] eval
    @ ./boot.jl:360 [inlined]
 [13] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:1116

My understanding is that the PyCall.jl is taking the ubuntu system python(libdynload) and thus
not able to import the numpy array since it is not installed on a system level but present at the
python virtual environment level.

How do I fix it so it runs? What can be alternative methods(if any) like maybe using Conda.jl I can
try, that will run the tutorial code?

The OSM Documentation on the lane tag (here) states, that this tag should be used to encode the total number of lanes (in both directions, if the way is oneway=false) available to motorized traffic. Looking at the defaults that have been chosen in LightOSM.jl, it seem to me like we think the lanes encodes only lines in one direction.
I think we should rework (and better document) this and all other tags which get set as a default, as well as the algorithm which sets the default for the lanes tag. Maybe, we could also parse the lanes:forward, lanes:backward and lanes:both_ways, to clean up any ambiguity?
I would be happy to take a shot at it, but the question of how to parse data, especially from OSM is always heavily opinionated, so I would like to get a little discussion started about how we would like to handle this problem, before I get stuck in.

Would be useful to have an argument in the astar function in traversal.jl that pre-specifies the number of nodes to traverse under the astar algorithm, and breaks/returns a 'no path found' if the traversal limit is reached without finding a path

The use case would be for when we just want to check if a short enough path exists between two given nodes, without incurring the full runtime if the path between two given nodes is really long. (And if we have to repeat this many times)

Say I want to look at a specific w::Way{Int64}. I can access w.nodes and get a vector of nodes that the way is made up of.
I wonder if there is a way to identify the start and end node of a given Way{Int64}?

The docstring lists :bounding_box as a download_method, but the method needs :bbox

I'm currently building a plotting package for LightOSM maps and in this process it has occurred to me that it would be handy to have one (possibly two) additional fields in the OSMGraph type.

1. index_to_way: a dictionary which maps the index of an edge of the underlying OSMGraph.graph to the OSMGraph.Way it represents.
2. sorted_edges: a vector whose indices correspond to OSMGraph.graph.edges and maps to the indices of its start and end nodes OSMGraph.graph.vertices. (This field could also be named edge_to_vertices or something like that.)

This is how I've currently implemented it for my needs:
https://github.com/fbanning/OSMMakie.jl/blob/77052b22853f57e03d3712094075c46195d6f987/src/recipe.jl#L35-L46

I think that index_to_way could be very handy to be made available to all users as a field of the OSMGraph type. However, I'm not so sure about sorted_edges because it's solely concerned with the node and edge indices of the underlying graph and doesn't work with any OSM data at all.

Would like to hear your opinions on these extensions of OSMGraph. If you're feeling positive about it, would you accept a PR if I make one?

Been noticing some odd behaviour on certain roads that are not tagged as oneway, but when you download the data it is being treated as oneway.

Way in question: https://www.openstreetmap.org/way/51812001#map=18/-37.71841/145.08229
Does not have the 'oneway' tag

But if you pull the graph
julia>g = graph_from_download(:point, point=GeoLocation([-37.71866,145.08154]), radius=1, metadata=false, save_to_file_location="test_oneway.osm", network_type=:drive_mainroads, download_format=:osm)

And check the highway tags:

julia> g.highways[51812001].tags["oneway"]
true

Checking the source .osm that's downloaded doesn't show the oneway=true either

Checking other ways nearby, don't have this issue
https://www.openstreetmap.org/way/157211030

julia>g.highways[157211030].tags["oneway"]
false

The lightOSM OSMGraph is an abstraction of the underlying graph, but it is missing a few of the normal methods of interacting with graph traversal.

A good enhancement would be implementing the inneighbours and outneighbours functions from LightGraphs interfaces to remove dependencies on these functions elsewhere. The functiuons can be wrappers of the functions to pull out the in and out neighbours of the OSMGraph.graph, and it would be good if there were dispatches to let you choose between the OSM node id and the vertex index of the graph.

Hello. I was wondering if I can load OSM data from a file--instead of downloading from the internet? I downloaded a big OSM file and I wanted to see if I could import that data into LightOSM for analysis. I did not see this file access stuff in the tutorial or api docs, but I imagine it must be in there somewhere :). Thanks.

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

In the extension pgRouting of PostgreSQL, there is a family of functions (withPoints) to calculate network distances between arbitrary points and not nodes by temporarily adding points on edges. This option is useful when edges are very long or when targeting short distances (e.g., walking distance).
Would it be feasible in LightOSM?
Using nearest_way is already better than nearest_ node, but the partial distance of the closest edge is not considered.
Thanks.

I've downloaded a graph close to Vienna (AT) using:

LightOSM.download_osm_network(:bbox; network_type=:drive, save_to_file_location="./graph_drive", download_format=:osm, minlat=48.15, maxlat=48.40, minlon=16.20, maxlon=16.58)

Loading this graph using

LightOSM.graph_from_file("graph_drive.osm"; graph_type=:simple_weighted, precompute_dijkstra_states=false)

takes a really long time (just compared to previous tries in osmnx and igraph), and finally fails with

ERROR: TypeError: in typeassert, expected Int8, got a value of type Int64

Stacktrace:
[1] _roundedmean at C:\Users\USER\.julia\packages\LightOSM\Da141\src\graph.jl:218 [inlined]
[2] add_node_tags!(::LightOSM.OSMGraph{Int32,Int64,Float64}) at C:\Users\USER\.julia\packages\LightOSM\Da141\src\graph.jl:230
[3] graph_from_object(::LightXML.XMLDocument; network_type::Symbol, weight_type::Symbol, graph_type::Symbol, precompute_dijkstra_states::Bool, largest_connected_component::Bool) at C:\Users\USER\.julia\packages\LightOSM\Da141\src\graph.jl:36
[4] graph_from_file(::String; network_type::Symbol, weight_type::Symbol, graph_type::Symbol, precompute_dijkstra_states::Bool, largest_connected_component::Bool) at C:\Users\USER\.julia\packages\LightOSM\Da141\src\graph.jl:84
[5] top-level scope at REPL[2]:1

Unfortunately I could not trace that any further. Any ideas?
Using: LightOSM v0.1.18, Julia Version 1.5.2

Introduce a stable API to build a graph from nodes, ways and restrictions. This is useful for unit tests amongst other features

Need a 0.18.0 entry for DataStructures compact

So, somehow the OSM building format has changed or something, since this error was not happening some weeks/months ago, and checking out an old version of LightOSM throws exactly the same error:

using LightOSM
download_osm_buildings(:bbox;
    area...,
    metadata=true,
    download_format=:osm,
    save_to_file_location="london_buildings.osm"
);
# load as Buildings Dict
buildings = buildings_from_file("london_buildings.osm");

ERROR: KeyError: key 5461892572 not found
Stacktrace:
 [1] getindex(h::Dict{Int64, Dict{String, Any}}, key::Int64)
   @ Base .\dict.jl:484
 [2] parse_osm_buildings_dict(osm_buildings_dict::Dict{String, Any})
   @ LightOSM C:\Users\sdani\.julia\packages\LightOSM\Ksg88\src\buildings.jl:215
 [3] buildings_from_object
   @ C:\Users\sdani\.julia\packages\LightOSM\Ksg88\src\buildings.jl:260 [inlined]
 [4] buildings_from_file(file_path::String)
   @ LightOSM C:\Users\sdani\.julia\packages\LightOSM\Ksg88\src\buildings.jl:294
 [5] top-level scope
   @ c:\Users\sdani\SimiWorld\ProgrammerLife\GeoDev\dev\Tyler\docs\examples\UserGuide\osmmakie.jl:28

I could work around by inserting an haskey into parse_osm_buildings_dict, but I didn't understand the code well enough to make sure it's the correct fix.

I wonder if we can use FastPriorityQueues.jl. The benchmarks for Dijkstra give a 10x improvement!

Currently struct OSMGraphs types.jl has 'highways' as the field, this should be refactored to 'ways' to handle more general types of ways (e.g. rail)

For backwards compatibility, @mmiller-max proposed to overload the getproperty for the struct, so that when users call g.highway under the hood g.ways will be called instead

Currently, the shortest_path function throws an error when there is no path from the source to the destination. This makes handling the case where the path doesn't exist require a try-catch block, which isn't good for performance. Can it be refactored to return nothing in such a scenario?

Need a reliable way to test this.

Hi,

does this package have notable performance improvements w.r.t. OSMX.jl, in loading data, creating the graph and/or computing distances or nearby nodes? Can you please add such a short discussion in the docs?

I think JSON3 could bring some speed improvements, and we could possibly make use of StructTypes to help read OSM graph straight into an OSMGraph, maybe

I think there's a use case for pulling down graphs that filter out different road ('way') types,

e.g. for filtering out residential roads, which would be handy if we want to download all the highways in a country, without blowing up the file size with all the residential roads.

Perhaps this could be implemented as a post-processing step that filters out the specific way types

Unfortunately this will be a breaking change, so we should aim to release it along with any others (if there are any in the pipeline)

If only one nearest node is requested, the output should not be a vector. Currently this is the output:

julia> nearest_node(g, 1)
([[2]], [[0.1]])

And I think the output should be

julia> nearest_node(g, 1)
(2, 0.1)

Then if more points are requested

julia> nearest_node(g, 1, 2)
([2, 3], [0.1,0.1])

And if more locations are given

julia> nearest_node(g, [1, 4], 2)
([[2, 3],[5, 6]], [[0.1,0.1],[0.1,0.1]])

Unfortunately that makes it type unstable. Could have nearest_node and nearest_nodes potentially?

To investigate: When using shortest_path, it seems that if a path is not found it throws a LoadError

g = graph_from_download(:point, point=GeoLocation([-37.6965,145.0937]), radius=5, metadata=true)
path = LightOSM.shortest_path(g, 27162110, 679051448, algorithm=:astar)  # this is a LoadError

However, when this is wrapped in a try-catch, the type of the error gets changed to a generic exception error?

try
    path = LightOSM.shortest_path(g, 27162110, 679051448, algorithm=:astar)
catch err
    if isa(err, LoadError)
        println(err)
    else
        println("Not a Load err", err)  # ExceptionError
    end
end

Hi, this package looks great. But much more than what I need. It would be nice have a package that just wraps overpass queries so we can do them from julia.

Any chance the downloads could be separated out into a sub-package and genrealised to any object type? (e.g. I need rivers...). This could be in the same repo but with only the HTTP dependency, and registered separately?

Hi, I'm not sure if I should post this here but I cannot open an issue here https://github.com/captchanjack/deck.gl/tree/pydeck/julia-pycall-binding/bindings where my issue more strictly resides.

I'm trying to follow your tutorial https://deloitteoptimalreality.github.io/LightOSM.jl/notebooks/tutorial/
but I'm having trouble building the pydeck bindings when calling yarn bootstrap

I've tried building at Pkg.dir("PyCall") and at Pkg.envdir("PyCall')
At both sites I've tried running

rm -rf node_modules
yarn cache clean
yarn
yarn bootstrap

I should also say that yarn run test fails with 85 error Unable to resolve path to module '@deck.gl/core' import/no-unresolved

Thanks in advance

With the archiving of LightGraphs and switching to Graphs (see here) we need to move to Graphs.jl. For this, we need all dependencies that used LightGraphs to have switched, which means we are now waiting on JuliaGraphs/StaticGraphs.jl#30 as MetaGraphs and SimpleWeightedGraphs have both been updated.

From https://github.com/DeloitteOptimalReality/LightOSM.jl/blob/master/benchmark/benchmarks.jl

point = GeoLocation(-37.8142176, 144.9631608) # Melbourne, Australia
radius = 5 # km
data_file = joinpath(@__DIR__, "benchmark_map.osm")

g_losm = LightOSM.graph_from_download(:point,
                                      point=point,
                                      radius=radius,
                                      weight_type=:distance,
                                      download_format=:osm,
                                      save_to_file_location=data_file)

# g_losm = LightOSM.graph_from_file(data_file, weight_type=:distance)
@time g_losm_precompute = LightOSM.graph_from_file(data_file, weight_type=:distance, precompute_dijkstra_states=true)

throws and UndefVarError blaming a parameter n in src/graph.jl:455 which simply initialises a vector to hold g.dijkstra_states = Vector{Vector{U}}(undef,n)
Shouldn't this be g.dijkstra_states = Vector{Vector{U}}(undef,length(g.nodes)) ?

Not sure why, probably something in the yaml