To my knowledge (very happy to be proven wrong).

QL could be an ideal language for representing complex queries in text I think: see QL Basics. Illustrative examples (I don't think we can currently do this - any thoughts):

  ["key"]            /* filter objects tagged with this key and any value */
  [!"key"]           /* filter objects not tagged with this key and any value */
  ["key"="value"]    /* filter objects tagged with this key and this value */
  ["key"!="value"]   /* filter objects tagged with this key but not this value */
  ["key"~"value"]    /* filter objects tagged with this key and a value matching a regular expression */
  ["key"!~"value"    /* filter objects tagged with this key but a value not matching a regular expression */
  [~"key"~"value"]   /* filter objects tagged with a key and a value matching regular expressions */
  [~"key"~"value",i] /* filter objects tagged with a key and a case-insensitive value matching regular expressions */

Questions:

• Any ideas how that would be rewritten in:
  • PostgreSQL/PostGIS?
  • osmdata?
• Would it be worth thinking about allowing osmdata to access data stored in a PostGIS db?
• Thinking: it would be great to put some of these example in vignettes / examples. Good idea?

E.g. with

hs <- issue_query (q2)
Error: could not find function "issue_query"

This commit removed most dplyr deps, but has left this one. Once final types & classes are established, try to think of use-cases where pipes would not work anyway to see if this dep is really necessary.
cc @Robinlovelace

What does devtools::load_all (".", export_all=FALSE) do? Is it needed?

@mpadge could you go to settings and make sure that the GitHub pages source is set to “master branch /docs folder”.? Thanks!

No idea what caused it, just reporting from newly installed version on R 3.3.3 on my Win desktop. Can anyone reproduce? Screenshot + code that caused it below:

Thanks @Robinlovelace for the introduction and hello @masalmon - great to finally e-meet you back. These are two brief Qs I just emailed to you via Robin:

1. You've got an Roxygen field here that says "@author Maelle Salmon". Are you okay with that line being removed, because:

   i. authorship for individual functions is not general R practice;

   ii. this raises difficult questions of whether anyone else has the right to accept a PR that modifies that function, and what the potential status of your authorship would be thereafter?; and

   iii. You're in the author list in DESCRIPTION anyway

You can have an additional description of what you contributed (like these), but if you're going to be involved further, it'd likely be better and more meaningful to simply list you as a general "author" absent specifications. I'd appreciate an indication of your preference in this regard.

2. Are you okay with the diacritical marks being removed from your name, because CRAN rejects those anyway? (It'll then just become ''Maelle'' - blame ASCII for that, as I'm sure you're well aware.)

@edzer Your help would be much appreciated on getting travis to pass with suggests:sf on osx as well as linux - my current travis.yml is here, with these travis fails for linux and osx. It'd also be great to load sf within a travis container, which would require sudo:false. Any assistance greatly appreciated!

Example, from which I was expecting to get Lovell park as a polygon as it is:

b = tmap::bb("Leeds")
q = opq(bbox = b) %>% 
  add_feature("leisure", "park")
p = overpass_query(query = q)
library(sp)
plot(p$osm_points)
mapview::mapview(p$osm_points)
plot(p$osm_polygons)

Instead I get just points of the border by the look of it:

Queries returning no data produce the following error:

> dat
$osm_nodes
[1] coordinates
<0 rows> (or 0-length row.names)

$osm_ways
Error in signif(sapply(coordinates(x), function(x) x[[1]][1, ]), digits = digits) : 
  non-numeric argument to mathematical function

and yet

> dat$osm_ways
An object of class "SpatialLinesDataFrame"
Slot "data":
[1] name   oneway type  
<0 rows> (or 0-length row.names)
...

and also

> sapply (dat, class)
                 osm_nodes                   osm_ways                 osm_polygons
  "SpatialPointsDataFrame"    "SpatialLinesDataFrame"   "SpatialPolygonsDataFrame"
> summary (dat)
             Length Class                    Mode
osm_nodes    0      SpatialPointsDataFrame   S4  
osm_ways     0      SpatialLinesDataFrame    S4  
osm_polygons 0      SpatialPolygonsDataFrame S4  

Because obviously print does not recognise the list item as sp. Most direct fix is likely to be creating a distinct class with its own print/summary method.

Solution: @masalmon myself and probably others will write one (Mark please digitally meet Maëlle).

Using the new reprex package for this: http://jennybc.github.io/reprex/

And linking to online plots like this:

Current tests for sf-construction contain a work-around for sf 0.4.3. These can be reverted on release of sf 0.5.0. Just search for 0.5.0 and uncomment.

From the README:

osmcsv <- '[out:csv(::id,::type,"name")];
area[name="Bonn"]->.a;
( node(area.a)[railway=station];
  way(area.a)[railway=station];
  rel(area.a)[railway=station]; );
out;'

obj <- overpass_query(osmcsv)
##  Error in eval(substitute(expr), envir, enclos) : expected < 

As discussed in this comment regarding sf, it would likely be very useful to have some kind of recursion operator for osmdata, along the lines of

> get_points (osmdat$osm_lines)

to return all data for all points comprising those lines, including additional key-val pairs for the points. This is an implicit ''recursion down'' operator. An analogous ''recursion up'' operator would be

> get_lines (osmdat$osm_points)

which would return all lines which contain those points.

A question worth thinking about:

• Are then any instances in which the recursion direction would not be obvious?

If no, then only three functions need be written, for lines, points, and polygons.

When building the package every time it creates new files under src/ but I guess it's normal?

Example:

origins = dl_origins(bb)
q = osmdata::opq(bbox = bb) %>% 
  osmdata::add_feature("place", "*")
origins = osmdata::overpass_query(q)
origins

$osm_lines
Error in signif(sapply(coordinates(x), function(x) x[[1]][1, ]), digits = digits) : 
  non-numeric argument to mathematical function

It's probably not a meaningful query but think the error message could be a little more informative at least!

While looking through the function reference, I saw that the page for opq_to_string is missing.

Not sure if this is possible already, if it is should be documented in the vignette.

Example: ropensci/osmplotr#4

This could be improved by providing concrete examples, particularly in the section on 'GDAL translation of OSM'. It would also be greatly improved by adding a second example other than routing. @Robinlovelace - any ideas here?

TODO: Replace all links to first vignette with URL of the webpage once that is up and running.

It works okay onscreen, but the vignettes show that having several consecutive message lines is not okay. Fix it!

I'm currently using the package in RStudio on Windows.

• When I test the package with testthat, everything works ok.

• I am attaching the results of R CHECK. The errors are due to examples in which there are comments about their not working so seems ok?
  osmdata_check_results.txt

• If I run this query in my R session

qry <- opq(bbox=c(-0.12,51.11,-0.11,51.12)) 
qry <- add_feature (qry, key='highway', value='tertiary')
res <- overpass_query (qry)

The session aborts not long after the message

Issuing query to OSM Overpass...
Rate limit: 2

I re-started and now I actually get an error instead of session abortion

> qry <- opq(bbox=c(-0.12,51.11,-0.11,51.12)) 
> qry <- add_feature (qry, key='highway', value='tertiary')
> res <- overpass_query (qry)
Issuing query to OSM Overpass...
2 slots available now.
[1] "2 slots available now."
 Show Traceback
 
 Rerun with Debug
 Error in make_query(query, quiet) : Bad Request (HTTP 400). 

It seems quite weird to me. I can't investigate further today, but I'll try to take a few minutes this week at work to install the package and run the same steps there (still Windows).

Just a flag for me to do that

This is what I get when running the code, running from https://github.com/cyipt/example-data

ct = osmdata_sp(doc = "../example-data/chapeltown-road/chapeltown.osm")
## q missing: osmdata object will not include query
## Error: cannot allocate vector of size 130331.5 Gb

For routing purposes, it'll be necessary to read an osmdata object and extract all ways as a single data.frame of [from_id, to_id, from_x, to_x, distance].

cc @karpfen

This should be done, notably in response to @edzer's ropensci review of the package. We just need to work out the best place / places to do it. cc @Robinlovelace @maelle

A question for @Robinlovelace and @hrbrmstr: How do we want the user to control what is returned from overpass? The current overpass_query relies, though process_doc, on the query being precise enough that it will only return the lowest desired OSM objects (node->way->rel), while osmdatar currently has the three functions get-points, get-lines, and get-polygons. The two approaches now need to be reconciled within osmdata, so I'll begin with an argument for my current approach:

1. Bob's overpass approach relies on the queries being very precise, which they may not always be.
   1. Example: multiipolygons, for which a user may understandably desire the underlying way members rather than the multipolygon relation, yet the extension of process_doc to include processing of polygons would return just the latter and not the former.
   2. Example: There may be situations in which someone actually desires all the nodes which make up a way (that is, SpatialPoints), rather than the ways themselves (SpatialLines). This is possible with the osmdatar approach, yet not the overpass approach.
2. Many OSM are represented in a variety of formats, regardless of how strict OSM guidelines are. Public transport stations are exemplary: Many may be either node or polygon objects, and simply asking for bus_station will return only those mapped as polygons, yet will miss all purely nodal stations.
   1. Example: The overpass approach (implicitly extended to polygon extraction) will not allow a map of bus_stations as simple points, because a query for bus_stations will return the highest hierarchical objects, which will be (polygonal) ways (or maybe even multipolygons).

Having stated the essence of my thoughts on the matter, I of course acknowledge that Bob's overpass approach is superior in one important way of obviating any need for users to specify a desired kind of output. This would be fantastic in a perfectly tidy OSM world, but I fear that the inherent messiness of everything actually requires the user to exert some degree of control, for which the simplest is surely specifying points, lines, or polygons. Thoughts?

These are permitted in OSM yet not by SF. osmdata.cpp needs to be modified to merge any touching inner rings into single rings. See also this extended OSM discussion.

Reproducible example:

q <- opq (bbox="Kunming, China")
osmdata_sf (q)

Error in rcpp_osmdata_sf(doc) : unknown member_type

But this works:

osmdata_sf (add_feature(q, key = "highway"))

Object of class 'osmdata' with:
                 $bbox : 24.8915153,102.5417638,25.2115153,102.8617638
        $overpass_call : The call submitted to the overpass API
            $timestamp : [ Sun Feb 26 23:29:43 2017 ]
           $osm_points : 'sf' Simple Features Collection with 50300 points
            $osm_lines : 'sf' Simple Features Collection with 5106 linestrings
         $osm_polygons : 'sf' Simple Features Collection with 65 polygons
       $osm_multilines : 'sf' Simple Features Collection with 0 multilinestrings
    $osm_multipolygons : 'sf' Simple Features Collection with 0 multipolygons

Is it because the query is simply too large?

osmium implements its own translation of OSM into Simple Features here - make sure that osmdata is compatible.

This would probably be quite useful, and easily added to opq (). Should probably also default to recurse down relation (>>).

As far as I see things, osmdata is now largely done and includes functionality necessary for initial release. Only remaining tasks:

1. @Robinlovelace finish vignette
2. @masalmon once Robin's done that, pkgbuild the website again
3. add examples to fn docs

@hrbrmstr plan now is to submit this first to ropensci to gain some hopefully useful feedback prior to CRAN submission. Any comments, suggestions on general package structure at this stage? Or just a general nod of approval for us to go ahead and submit with you as package co-author?

Thanks all for all the help!

See lots of discussion in #31. The problem is particularly pronounced in appveyor builds, but also plagues travis. Both overpass and nominatim are prone to timing out, which leads to tests failing.

Current approach for overpass is in the overpass_status function, which tries 10 times with a 100second timeout, yet still occasionally fails. I've also implemented a URL status check for nominatim called url_available.

Tests seem generally okay now (and even if they weren't, could technically be made okay with if clauses to perform tests only when timeout not reached), yet building vignettes still occasionally fails. This latter can not be avoided through if !timeout trickery, and so will always risk failing CRAN checks.

This problem must be solved ...

cc @masalmon

osmdata currently uses the generic rapidxml parser. Write a dedicated parser to speed things up.

Sometimes you just want to download osm data and save it as a .osm file, e.g. to open in QGIS or JOSM. Is this possible?

Next request for cognitive input from @Robinlovelace and @hrbrmstr:

How are we going to construct overpass queries? Some food for thought in opq.R here, in which I've implement an option for non-exact matches. How far do we go aiding a user in avoiding the need to understand regex? I'd suggest a few simple extensions might actually go a long way in that regard, thereby making it easier to use for more people, but likely at the expense of requiring several additional arguments for add_feature.

A further issue: Many queries may not be constructed through sequential addition - prime example being AND rather than OR queries, because AND queries are not sequentially independent. (Consider that the example given in the function opq here actually represents an OR-construction rather than an AND-construction.) And yet only OR queries are possible in the structure both of add_feature and the osmdatar approach of specifying extra_pairs (see lines 60-69 here).

I'd suggest that in addition to non-exact values, we could possibly implement further options for:

1. Negation
2. Match case or not? (This will be messy to implement, but possible)
3. Match position (maybe 0=start, 1=end, any other value = anywhere)
4. Accented characters - maybe could avoid an extra arg by simply replacing any such with wildcards?

This would leave only the task of dealing with proper AND queries, which requires a different approach to that of add_feature, which is unavoidably OR. Thoughts? Suggestions? Food for thought from this point of the overpass language API onwards.

See comments on commits here and here. Solution is from @hrbrmstr : Insert osm_search into osmdata.

@Robinlovelace I've now included this benchmarking file, but with the sf code commented out. Travis fails otherwise (see here) because sf requires all of this pre-install stuff. We can easily put that in later, but in the meantime to keep our travis build times relatively short, we can do without it. This issue is just a flag to fix that up later.

@Robinlovelace here's one for ya: This should crash at some stage in the following way:

for (i in 1:100) q <- opq ("leeds")
# Error: lexical error: invalid char in json text.
#                                        <!DOCTYPE HTML PUBLIC "-//IETF/
#                      (right here) ------^

Could you catch these errors and spit out a simple "sorry, nominatim is broke; please try again" message?

This is unuseful, e.g.:

Robinlovelace@dad2fab#diff-c0aac139f53cce21e172492141a82bf3R465

Note I was forced to use as.character() - better if the data is stored as chars IMO.

The whole package is currently structured around sp objects, with overpass_query returning an osmdata object embedded within which are three sp objects (points/lines/polygons). How can sf-class objects best be incorporated? Possibilities:

1. Have overpass_query return some more generic object the members of which can be accessed via:

   (i) sp_osm_points(), sp_osm_lines(), sp_osm_polygons(); and

   (ii) sf_osm_points(), sf_osm_lines(), sf_osm_polygons()

2. Keep things as they are and include an osmdata_to_sf function which will convert all sp members to sf, either by:

   (i) Simply including an sf dep - a massive increase in package size and complexity!, or

   (ii) Hard-coding sf construction within the Rcpp routines as currently done for sp

Thoughts @Robinlovelace ?

Write them, and consider this issue.

Many potential solutions - thoughts?

Robin, This is something you could definitely help with! I'll take care of the C++ coverage, but you could definitely extend the R tests. One big advance will be an OSM example with a multipolygon relation (that is, with inner and outer components), so if you can dig out a simple one for that, that'd be grand. That'd also be ideal for me to address #20.

(I don't think anything can be done about `stub.R, but will give that a ponder myself ...)

By area (meters squared) would be good too.

• Add CI on OSX via Travis? E.g. I do it here

• Add CI on Windows via Appveyor?

In response to @edzer's ropensci review of the package, it's be useful to implement a function to enable the standard osmdata output, which simply contains the entire data of all geometric types, to only those components of each geometric class that are not present in other classes. The result of this function would then be more comparable with the output of GDAL. cc @Robinlovelace @maelle