yonghah / esri2sf Goto Github PK

Hi! I'm trying to run a request on a table that I've manually queried before, but esri2df is telling me that it's not a valid URL and then throwing an error.

rezz = esri2df('https://arc.indiawris.gov.in/server/rest/services/SubInfoSysLCC/WaterResourceProject/MapServer/41', objectIds = paste(1:50, collapse = ","))
Error in value[[3L]](cond) : Url is not a valid ESRI Service Url.
Could not access url with {httr}.
In addition: Warning messages:
1: In request_perform(req, hu$handle$handle) :
  restarting interrupted promise evaluation
2: In request_perform(req, hu$handle$handle) :
  internal error -3 in R_decompress1

I believe this is a bug, I may have introduced in #29. I will be working on a new pull request, but TLDR is that I believe listing a large number of objects ids in a single query request results in an error in the response.

If I try to pull the layer I get kind of a weird and unexpected response as I know this layer has features:

out <- esri2sf::esri2sf(url = 'https://services.arcgis.com/8ZpVMShClf8U8dae/arcgis/rest/services/TestingLocations_public2/FeatureServer/0')
#Layer Type: Feature Layer
#Geometry Type: esriGeometryPoint
#Service Coordinate Reference System: 3857
#Output Coordinate Reference System: 4326
out
#Simple feature collection with 0 features and 0 fields
#Bounding box:  xmin: NA ymin: NA xmax: NA ymax: NA
#Geodetic CRS:  WGS 84
#[1] geoms
#<0 rows> (or 0-length row.names)

Show max record count for url.

url <- "https://services.arcgis.com/8ZpVMShClf8U8dae/arcgis/rest/services/TestingLocations_public2/FeatureServer/0"
queryUrl <- paste(url, "query", sep = "/")
esri2sf:::getMaxRecordsCount(queryUrl, token = "")
#[1]  50000

Show record count

getRecordsCount <- function(queryUrl, where, token = "", ...) {
  
  query <- list(where = where, returnCountOnly = 'true', token = token, f = "json", ...)
  
  responseRaw <- content(POST(queryUrl, body = query, encode = "form",
                              config = config(ssl_verifypeer = FALSE)), as = "text")
  response <- fromJSON(responseRaw)
  
  response[['count']]
  
}

getRecordsCount(queryUrl = "https://services.arcgis.com/8ZpVMShClf8U8dae/arcgis/rest/services/TestingLocations_public2/FeatureServer/0/query",
                where = '1=1',
                token = "")
#[1] 42438

Because the max record count is greater than the total number of records. The way the esri2sf function will work (or more specifically the esri2sf:::getEsriFeaturesByIds() function) it will list all the object ids for all 42,438 records in the query url. I think this is hitting some sort of max url length of some sort sending back an unaccounted for error. This can be shown with the following code where I step through the interior functions in the package.

#++++++++++++++++++++++++++++++++++
# Inputs to esri2sf function
#++++++++++++++++++++++++++++++++++
url = "https://services.arcgis.com/8ZpVMShClf8U8dae/arcgis/rest/services/TestingLocations_public2/FeatureServer/0"
outFields = c("*")
where = "1=1"
bbox = NULL
token = ""
geomType = NULL
crs = NULL
#++++++++++++++++++++++++++++++++++


#++++++++++++++++++++++++++++++++++
# Part of the esri2sf funciton
#++++++++++++++++++++++++++++++++++
layerInfo <- fromJSON(content(POST(url, query = list(f = "json",
                                                     token = token), encode = "form", config = config(ssl_verifypeer = FALSE)),
                              as = "text"))
message(paste0(blue("Layer Type: "), magenta(layerInfo$type)))
if (is.null(geomType)) {
  if (is.null(layerInfo$geometryType))
    stop("geomType is NULL and layer geometry type ('esriGeometryPolygon' or 'esriGeometryPoint' or 'esriGeometryPolyline') could not be inferred from server.")

  geomType <- layerInfo$geometryType
}

message(paste0(blue("Geometry Type: "), magenta(geomType)))
# Layer Type: Feature Layer

if (!is.null(layerInfo$extent$spatialReference$latestWkid)) {
  layerCRS <- layerInfo$extent$spatialReference$latestWkid
} else if (!is.null(layerInfo$extent$spatialReference$wkid)) {
  layerCRS <- layerInfo$extent$spatialReference$wkid
} else if (!is.null(layerInfo$extent$spatialReference$wkt)) {
  layerCRS <- layerInfo$extent$spatialReference$wkt
} else {
  stop("No crs found. Check that layer at url has a Spatial Reference.")
}
message(paste0(blue("Service Coordinate Reference System: "), magenta(layerCRS)))
# Geometry Type: esriGeometryPoint

if (class(bbox) == "bbox") {
  if ((st_crs(bbox)$input != layerCRS) && !is.null(layerCRS)) {
    bbox <- st_bbox(st_transform(st_as_sfc(bbox), layerCRS))
  }
} else if (!is.null(bbox)) {
  stop("The provided bbox must be a class bbox object.")
}

bbox <- paste0(unlist(as.list(bbox), use.names=FALSE), collapse = ",")

queryUrl <- paste(url, "query", sep = "/")

# esriFeatures <- getEsriFeatures(queryUrl, outFields, where, bbox, token, crs, ...)
#++++++++++++++++++++++++++++++++++

#++++++++++++++++++++++++++++++++++
# Step into the source code for esri2sf:::getEsriFeatures()
#++++++++++++++++++++++++++++++++++
fields = outFields # added to make step through easier
ids <- esri2sf:::getObjectIds(queryUrl, where, bbox, token)
if (is.null(ids)) {
  warning("No records match the search criteria.")
  return()
}
maxRC <- esri2sf:::getMaxRecordsCount(queryUrl, token)
idSplits <- split(ids, seq_along(ids) %/% maxRC)
# Because maxRC = 50000 and the number of records is 42438, idSplits is
# a list of length 1 with a vector of objects ids with length 42438.
if (is.null(crs)) {
  crs <- ""
} else if (is.numeric(crs)) {
  crs <- as.character(crs)
} else if (isWktID(crs)) {
  crs <- sub(pattern = "^(EPSG|ESRI):", replacement = "", x = crs)
} else {
  crs <- toJSON(list("wkt" = WKTunPretty(st_crs(crs)$WKT1_ESRI)), auto_unbox=TRUE)
}
# results <- lapply(idSplits, getEsriFeaturesByIds, queryUrl, fields, token, crs, ...)
#++++++++++++++++++++++++++++++++++


#++++++++++++++++++++++++++++++++++
# Step into esri2sf:::getEsriFeaturesByIds()
#++++++++++++++++++++++++++++++++++
ids = idSplits[[1]] # added to make step through easier
query <- list(objectIds = paste(ids, collapse = ","),
                outFields = paste(fields, collapse = ","),
                token = token, outSR = crs, f = "json")#, ...) #removed because ... only works in function

responseRaw <- content(POST(queryUrl, body = query, encode = "form",
                              config = config(ssl_verifypeer = FALSE)), as = "text")

response <- fromJSON(responseRaw, simplifyDataFrame = FALSE, simplifyVector = FALSE,
                       digits = NA)

#response$features #the function usually returns an object with a 'features' index, but in this case it is only returning an object with and error index.
response #Added to show issue
# $error
# $error$code
# [1] 400
# 
# $error$message
# [1] "Cannot perform query. Invalid query parameters."
# 
# $error$details
# $error$details[[1]]
# [1] "Unable to perform query. Please check your parameters."
#++++++++++++++++++++++++++++++++++

What this means is that the response object doesn't contain the actual response features because an error is occurring. What I will probably propose in my following pull request is a switch to look for an error in the response and if detected throw a stop() and passthrough the error message. In this case its not terribly informative but it at least prevents the return of a a null dataframe with no indication that something went wrong. I will also do some tests to see what is causing the issue at little bit deeper level and see if setting some upper limit to the max record count will fix the issue.

I think there it may be useful to allow users to return a list of all folders and services hosted by an ArcGIS server. I came up with a function (tentatively named esriIndex although esriStructure could also work) that does this:

esriIndex <- function(url, parent = NULL, recurse = TRUE) {
  urlServer <-
    stringr::str_extract(url, ".+rest/services")
  
  urlInfo <-
    jsonlite::read_json(paste0(url, "?f=json"), simplifyVector = TRUE)
  
  folders <-
    tibble::tibble(
      name = urlInfo[["folders"]] |> as.character(),
      type = "Folder",
      url = paste0(urlServer, "/", urlInfo[["folders"]]),
      parent = parent
    )
  
  services <-
    tibble::tibble(
      name = urlInfo[["services"]][["name"]] |> as.character(),
      type = urlInfo[["services"]][["type"]] |> as.character(),
      url = paste0(urlServer, "/", urlInfo[["services"]][["name"]], "/", urlInfo[["services"]][["type"]], recycle0 = TRUE),
      parent = parent
    )
  
  urlIndex <-
    dplyr::bind_rows(
      folders,
      services
    )
  
  if (recurse) {
    if (length(folders[["name"]]) > 0) {
      urlIndex <-
        dplyr::bind_rows(
          urlIndex,
          purrr::pmap_dfr(
            folders,
            ~ esriIndex(url = ..3, parent = ..1)
          )
        )
    }
  }
  
  urlIndex
}

esriIndex(url = "https://carto.nationalmap.gov/arcgis/rest/services")
#> # A tibble: 11 × 4
#>    name                      type           url                           parent
#>    <chr>                     <chr>          <chr>                         <chr> 
#>  1 Utilities                 Folder         https://carto.nationalmap.go… <NA>  
#>  2 contours                  MapServer      https://carto.nationalmap.go… <NA>  
#>  3 geonames                  MapServer      https://carto.nationalmap.go… <NA>  
#>  4 govunits                  MapServer      https://carto.nationalmap.go… <NA>  
#>  5 map_indices               MapServer      https://carto.nationalmap.go… <NA>  
#>  6 selectable_polygons       MapServer      https://carto.nationalmap.go… <NA>  
#>  7 structures                MapServer      https://carto.nationalmap.go… <NA>  
#>  8 transportation            MapServer      https://carto.nationalmap.go… <NA>  
#>  9 Utilities/Geometry        GeometryServer https://carto.nationalmap.go… Utili…
#> 10 Utilities/PrintingTools   GPServer       https://carto.nationalmap.go… Utili…
#> 11 Utilities/RasterUtilities GPServer       https://carto.nationalmap.go… Utili…

^{Created on 2021-11-28 by the reprex package (v2.0.1)}

A couple of notes:

• esriUrl_isValid only validates MapServer and FeatureServer URLs so the url cannot be passed to esrimeta() without returning an error and the base url can't be extracted by the esri2sf::esriUrl_parseUrl() function for the same reason. It may be helpful to allow these functions to accept server or folder URLs.
• I'd love to add the option of returning a list of layers for all services as well as the folders and services. I think this should be relatively straightforward with the esriLayers() function but I ran into some performance issues when testing it out. Suggestions on how to incorporate this are welcome.
• The original esri2sf and esri2df functions used the convention of all lowercase function names. I noticed that more recent additions use a camel case convention that matches the conventions of ArcGIS web services. I stuck with the latter in this draft but wasn't sure if all lowercase may be preferred.

Amazing package developers!

More of question and not necessarily an issue but was curious how I would go about calling features behind a common access card (CAC) authentication? Is there a way to use the token argument?

I just added a new function to my esri2sf fork that works with the Search endpoint of the Portal Directory API (aka Sharing API) called esrisearch().

I haven't added any corresponding tests yet but I did add examples and, overall, it seems to work pretty well. If it isn't out of scope for this package, it may be useful to identify any other endpoints from this API that may be helpful to support in order to ensure consistency across the different possible functions.

As of now, the outSR parameter in the query list creation is hard coded to be WGS1984 EPSG:4326. I am guessing this is legacy before the crs argument was added esri2sf. I believe this needs updated so that the output CRS can be specified correctly.

I am going to fork the repository and do some testing and may upload a pull request.

Hello and thanks for this nice package!
I wanted to use it to access a specific ArcGIS REST server to get a polygon shapefile: ArcGIS REST

The polygons present in this server can be seen in this map: Map of Polygons

I used this code to download the third layer in the FeatureServer. I runs without error.

url <-"https://services.arcgis.com/XG15cJAlne2vxtgt/ArcGIS/rest/services/CRC_Community_Layers_May_2017/FeatureServer/3" community_sf <- esri2sf(url, crs = 3857)

The resulting sf object has missing features. I exported it using the following code:
st_write(community_sf, paste0(path_to_files,"eligible_communities_2017_3"), driver = "ESRI Shapefile", append = FALSE )

And loaded it into QGIS in which I accessed the same layer trough that all the polygons were present. Any idea why this is happening?

The ArcGIS REST API Query (Feature Service/Layer) operation supports spatial filters which would be a wonderful feature to add to this package. Here is the relevant section of the documentation:

The geometry to apply as the spatial filter. The structure of the geometry is the same as the structure of the JSON geometry objects returned by the ArcGIS REST API. In addition to the JSON structures, you can specify the geometry of envelopes and points with a simple comma-separated syntax.

Syntax:

JSON structures: geometryType=&geometry={ geometry}
Envelope simple syntax: geometryType=esriGeometryEnvelope&geometry=,,,
Point simple syntax: geometryType=esriGeometryPoint&geometry=,

I started working on this myself but ran into some trouble so I figured I'd create an issue here in case anyone else wanted to give it a try. Ideally, you could pass the results of the sf::st_bbox function to esri2sf rather than a string.

I use the bbox parameter of esri2sf all the time but I just wrote a get_esri_data() helper function that wraps esri2sf in my new {overedge} package and realized that it may make sense to pull one of the features into the main package.

Drawing on the example of the mapboxapi package, I set up the function so it can accept an sf object (as well as a bounding box) and then apply a buffer before creating the bounding box to pass into the query. But, the ArcGIS FeatureServer query API actually supports a distance parameter in combination with the data passed to geometry as a spatial filter. Ideally, implementing a location parameter could also include:

• Support for point as well as polygon sf data
• Access to the spatial relation options for spatial filtering. Those include: "esriSpatialRelIntersects | esriSpatialRelContains | esriSpatialRelCrosses | esriSpatialRelEnvelopeIntersects | esriSpatialRelIndexIntersects | esriSpatialRelOverlaps | esriSpatialRelTouches | esriSpatialRelWithin"
• Support for setting the units of the distance buffer

I'm currently doing all of this without the API using a wrapper function for sf_buffer but I'm assuming it may be faster for all that processing to take place on the server side. Happy to start working on this once the URL validation branch gets incorporated back into the package.

@elipousson Would you be able to give more details about this comment:

• I'd love to add the option of returning a list of layers for all services as well as the folders and services. I think this should be relatively straightforward with the esriLayers() function but I ran into some performance issues when testing it out. Suggestions on how to incorporate this are welcome.

What kind of performance issues did you get? Were there unexpected errors in the R code or was the server returning error codes? I was trying to test out what the issue could be but I have no examples of the issue you were having. Just as a note the '/layer' subpage is only available for MapServer and FeatureServer urls. If a feature url for one of these services is provided it will truncate the url to the service type.

#All Valid entries
esriLayers("https://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Demographics/ESRI_Census_USA/MapServer/3")
esriLayers("https://sampleserver1.arcgisonline.com/ArcGIS/rest/services/TaxParcel/AssessorsBasemap/MapServer")
esriLayers("https://carto.nationalmap.gov/arcgis/rest/services/contours/MapServer")

#Not Valid (current return shown below)-but better error handling (explicit check for '/(FeatureServer|MapServer)/?$' needs added)
esriLayers("https://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Elevation/ESRI_Elevation_World/GPServer")
# $error
# $error$code
# [1] 400
# 
# $error$message
# [1] "Unable to complete  operation."
# 
# $error$details
# [1] "GPTask 'layers' does not exist or is inaccessible."
# 
# 

Originally posted by @jacpete in #39 (comment)

Due to changes in how sf handles coordinate systems, your example code now produces a warning:

> library(esri2sf)
> url <- "https://services.arcgis.com/V6ZHFr6zdgNZuVG0/arcgis/rest/services/Landscape_Trees/FeatureServer/0"
> df <- esri2sf(url)
Linking to GEOS 3.8.0, GDAL 3.0.4, PROJ 7.0.0

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

[1] "Feature Layer"
[1] "esriGeometryPoint"
Warning message:
In CPL_crs_from_input(x) :
  GDAL Message 1: +init=epsg:XXXX syntax is deprecated. It might return a CRS with a non-EPSG compliant axis order.

I'm going to submit a small PR to circumvent this.

I just started prototyping this on this pull request #50 but I'm happy to pull the new esri2rast() function into a feature branch and remove it from the pull request whenever. Basically, I'd like to add support for ImageServer urls using the terra package.

I got a rough prototype working with the ExportImage API end point but the WMTS, the Export Tiles, or Raster Image APIs could all be alternate ways of accomplishing the same thing. Happy to work more on this or take suggestions on the best way to approach it!

I have been trying to extract tables from a MapServer without any success. Is it possible to make changes to the function to extract tables?

Thank you very much for this package. It is very interesting.

I'm fairly new at ArcGIS Online, but I was wondering: wouldn't it be easier (less code) to request geojson output from a hosted feature service?

For example:

https://services.arcgis.com/V6ZHFr6zdgNZuVG0/arcgis/rest/services/Landscape_Trees/FeatureServer/0/query?where=1%3D1&f=geojson

https://services.arcgis.com/V6ZHFr6zdgNZuVG0/arcgis/rest/services/Florida_Annual_Average_Daily_Traffic/FeatureServer/0/query?where=1%3D1&f=geojson

Of course this wouldn't work for a map service.

I'm not sure if this is essential but I'd be curious to see a comparison between how esri2sf provides access to the FeatureServer and MapServer REST API and how geosapi provides access to any GeoServer API. It is possible there are also some features for one or the other that could be helpful to emulate.

As discussed previously, this is the issue to support discussion of a forthcoming pull request to switch the package over from {httr} to {httr2}. Simultaneous with making these changes, I also made several other significant (more or less) changes:

• Switched over to using {cli} for almost all messages and alerts (I left addDomainInfo.R and token.R alone)
• Added a geometry parameter to esri2sf() to begin the process of allowing other types of spatial filters (not just bounding boxes)
• Added the new esriIndex() function described in #39
• Added a new esriInfo() function that includes a sitemap option that provides similar (but more limited) functionality as esriIndex
• Reorganized the code since zzz.R was getting very crowded (I split off esri2sfGeom.R and esriLayers.R)

A few things to highlight:

• The switch to {cli} should enable a variety of interface improvements. For example, I added the layer name to the default messages when calling esri2df() and esri2sf().
• I also added checks that attempt to download a layer with esri2sf if it fails with esri2df (and vice-versa).
• Using httr2 should make it easier to build new functions that support more elements from the ArcGIS REST API in the future. For example, esriInfo() was very easy to create after I had esriRequest() set up.
• The new geometry parameter can wholly replace the bbox parameter. It currently only supports bbox and POINT sf objects. Input geometry would need to be converted to the JSON format (e.g. esriGeometryPolygon) so I'm hoping that is easy to add later on.

I noticed a few issues that should be resolved before merging the pull request:

• I think error handling should be handled after making the request not as part of the URL validation. httr2 returns HTTP error codes but not much helpful detail.
• I think the path parameter for httr2::req_perform() may be helpful to get the package to work better with larger requests (I noticed it was failing with some larger requests that it makes just fine right now)
• I'd love to get the progress bar working with cli::cli_progress_along(). I got this working briefly but it was failing in some cases so I dropped it and kept pbapply.
• The addDomainInfo function kept kicking back errors but I'm not sure if that was a pre-existing issue or if I introduced a new bug somehow.

I expect there are a few other changes or bugs that I'm forgetting to mention but hopefully this is enough to get a review started.

I am adding in a copy of my findings after working on #25 below. Essentially without requiring the user to download and install an external copy of GDAL (outside of the the binaries that ship with/ or are linked to with the {sf} package or trying to package it with this library, I think the solution will be to do a feature request to the {sf} package to get an interface to convert any GDAL compatible CRS to an ESRI WKT. When I get the issue made for {sf} I will link it below so that we can track it. I will probably try to tinker with doing the pull request for the {sf} package myself, but I have never worked with C++ before so it will probably be quite the learning curve. In the meantime the esri2sf() crs argument can only accept WKTid's unless anyone has a different solution.

So I've been working on a pull request for this issue and I am hitting a small snag. For the outSR parameter in getEsriFeaturesByIds() ESRI's query format (https://developers.arcgis.com/rest/services-reference/enterprise/query-feature-service-layer-.htm) requires the input to be either a WKTid or spatial reference JSON object that can be formatted to contain an ESRI wkt. In my testing and research, the ESRI wkt is different than the normal OGR WKT generally given by R, but GDAL does have the power to convert to it. What I am looking for now is a way to interface with the GDAL library linked in R through the {sf} package. On linux this is pretty easy because GDAL is compiled and installed separately from the sf install, but on Windows (and I think OSX but I don't run this OS to test) the GDAL library is compiled and installed with the {sf} binaries and it makes it difficult to interface with the GDAL library directly.

Without a way to link to GDAL, we will be limited in our inputs for the crs argument. We will have to restrict it to only using WKTids These can be either EPSG or ESRI authority WKTids though as the ESRI query format seems to be able to handle either without direct specification. However, sf::st_sf() does require direct specification so I added a helper (hidden) function getWKTidAuthority to do a direct query to the proj.db SQLite database in the PROJ library linked to by the {sf} package. This function adds imported dependencies for the {DBI} and {RSQLite} packages which were imported or suggested by {sf} already.

https://github.com/jacpete/esri2sf/blob/62c6e8a3368d1fe9bfe24e400110d5e8606e3fe8/R/zzz.R#L149-L172

To extend the function to be able to accept any crs format you can generally use in R (e.g., proj4string, WKT, numeric EPSG) there will need to be an interface directly to the GDAL gdalsrsinfo command line tool (I think it may be easiest to create and issue in {sf} to extend sf::gdal_utils() for this one; https://gdal.org/programs/gdalsrsinfo.html) or improve the {Rccp} interface to exportToWkt C++ API (https://gdal.org/api/ogrspatialref.html#_CPPv4NK19OGRSpatialReference11exportToWktEPPcPPCKc) with an option to specify the FORMAT parameter and create an exported R function to control it. While I think the implementation of the first option may allow slightly easier access to the functionality, I think the addition of the second may already be being discussed by package authors (see comment on line 115 of code from {sf} below.

https://github.com/r-spatial/sf/blob/405ca1ca4ebdbd5672948aea88251e5d12004740/src/gdal.cpp#L115-L128

Originally posted by @jacpete in #25 (comment) #25

Hi @yonghah,

Since you are using the following line :

You should add rstudioapi in the import filed in DESCRIPTION.

Thanks for your work on this!

The authentication with Arcgis ONnine has an other way to get a token, by using a clientId and clientSecret and uses a fixed server name

Explained here:
https://developers.arcgis.com/documentation/core-concepts/security-and-authentication/accessing-arcgis-online-services/

The following curl works for me and returns a working token:

 curl -X POST   
  -d "f=json"    
 -d "client_id=xxxxx"   
  -d "client_secret=yyyyy"    
 -d "grant_type=client_credentials"    
 -d "expiration=1440" 
https://www.arcgis.com/sharing/rest/oauth2/token/

I think we should have support for this here.

This issue is a place to discuss the standardization of function names within the package so future development will have some standards to work off of.

This was rightfully brought up by @elipousson in #39 (comment) with:

• The original esri2sf and esri2df functions used the convention of all lowercase function names. I noticed that more recent additions use a camel case convention that matches the conventions of ArcGIS web services. I stuck with the latter in this draft but wasn't sure if all lowercase may be preferred.

And I responded with #39 (comment):

This is definitely been influenced by my personal taste. I tend to mix camelCase with underscores being used for separating parts of a name. For example the esriUrl_ prefixed functions all deal with the same type of data (ESRI url strings) and then the actual function name is base of the full function name (esriUrl_isValid, esriUrl_parseUrl). I know this doesn't agree with the original esri2sf function that you rightly noted was all lowercase, but it does follow the camelCase of the functions in the zzz.R file that were there before I started adding stuff to this repo. Totally up for debate on what we decide and use as standards moving forward, just want to make sure we don't change too much of pieces that already exist as exported functions so we don't break peoples existing code (we can create aliases and use deprecation warnings if we do end up changing an exported function: see here for an example). My current (biased) suggestion is to use camelCase for new functions except for the core original function esri2sf and its direct counterpart esri2df. My main reason for avoiding lowercase names is readability but the '2' in these function names solves that issue. @yonghah feel free to chime in on any thoughts you have here as well. I am happy to keep adding functionality and fixing bugs in the package but when it comes to stylistic decisions I want to make sure your'e on board as well.

I wanted to make this it's own prominent issue for us all to discuss and decide on the best path forward.

Do you have any recommendations on how to get data from this server:

url <- "https://gis.blm.gov/arcgis/rest/services/hydrography/BLM_Natl_AIM_AquADat/MapServer"
df <- esri2sf(url)
NULL
NULL
Error in if (geomType == "esriGeometryPolygon") { : 
  argument is of length zero
In addition: Warning message:
In getEsriFeatures(queryUrl, outFields, where, token) :
  No records match the search critera

It sure seems like your package would be perfect, but I don't know enough about ArcGIS to know why it's coming back with an error.

I've tried a few iterations of the url (for example, my first guess was):

url <- "https://gis.blm.gov/arcgis/rest/services/hydrography/BLM_Natl_AIM_AquADat/MapServer/0"
df <- esri2sf(url)

It looks like the new URL checks added to this package may be too strict. With the current version of esri2sf, I'm getting the following error for this MapServer URL: https://egisdata.baltimorecity.gov/egis/rest/services/CityView/Liquor_Licenses/MapServer/0

Error in value[[3L]](cond): 
Url is not a valid ESRI Map or Feature Service Url.
Could not access url with {httr}.

I confirmed that the URL works fine with the version of esri2sf that preceded the addition of the esriUrl_isValid() function.

@jacpete - would you mind taking a look at this since you contributed the new functionality? Thanks!

I recently realized that the ESRI FeatureServer and MapServer metadata occasionally includes alias names for the fields that can include helpful explanatory information that may not be clear based on the original column names alone.

I think it could be helpful to take the fromJSON call that appears in both esri2sf and esri2df and turn it into a public function (I thought esrimeta seemed consistent with the other function names but I'm happy for better options) that allows users to return all layer info or return the fields alone. Let me know if this idea seems like a good addition to the package and I'd be happy to make a pull request. I included an example MapServer layer below that shows how field aliases may be used.

esrimeta <- function(url, token = "", fields = FALSE) {
  layerInfo <- jsonlite::fromJSON(
    httr::content(
      httr::POST(
        url,
        query = list(f = "json", token = token),
        encode = "form",
        config = httr::config(ssl_verifypeer = FALSE)
      ),
      as = "text"
    )
  )

  if (fields) {
    return(layerInfo$fields)
  } else {
    return(layerInfo)
  }
}

url <- "https://maps2.dcgis.dc.gov/dcgis/rest/services/DCOZ/Zone_Mapservice/MapServer/13"

esrimeta("https://maps2.dcgis.dc.gov/dcgis/rest/services/DCOZ/Zone_Mapservice/MapServer/13", fields = TRUE)

Hi! I've just recently taken over some code and have been having trouble running it. The code pulls data from this GIS website

https://services6.arcgis.com/ubm4tcTYICKBpist/arcgis/rest/services/BC_Flood_Advisory_and_Warning_Notifications_(Public_View)/FeatureServer/0

and is formatted to take the “Advisory” field (type: esriFieldTypeDouble) and then later use that number to plot flood advisory levels with ggplot.

The previous owner and I have the exact same code, but when I run it it returns this error:

For the previous owner the function pulls a number,

but for myself it pulls a character string (these are from different instances so the data isn't expected to match up here).

From the way the website describes the field it seems like it should return a number rather than a string, so it is weird that it would do that for one of us but not the other. I'm very new to R so this could just be something I'm doing wrong, but another more senior colleague has a similar bit of code that also pulls a string. His is formatted to accommodate this different data type and I could do the same, but any insight you might have as to why this might happen would be greatly appreciated!

Some server admins disable the ArcGIS Server Services Directory for a bit of security through obscurity and set credentials on services (and, if Services Directory is not disabled, may also hide folders from viewing unless the user has provided the proper credentials.)

esriUrl_isValid fails in this case with a 403 status. e.g.

Response [https://www.redacted_server_name.com/arcgis/rest/services/folder_name/service_name/FeatureServer/1]
  Date: 2022-01-21 05:10
  Status: 403
  Content-Type: text/html;charset=utf-8
  Size: 630 B
<html lang="en">
<head>
<title>
Error: Services Directory has been disabled.</title>
<link href="/arcgis/rest/static/main.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<table width="100%" class="userTable">
<tr>
<td class="titlecell">ArcGIS REST Framework</td>

One Solution:
Appending "?f=json" to the feature layer or feature service url being checked will return a status_code of 200 and the body of the response will be {'error': {'code': 499, 'message': 'Token Required', 'details': []}}
It also works if the url is shortened back to the instance.

You would think that just adding the token (with paste0("?token=", token) would solve this, but doing this still returns the 403 error.

The modification would be at the top of the function (can use a GET or POST, both seem to work fine against my server).

# check url succeeds (url without ""?f=json"" will return a status code 400 for 'hidden' services/layers)
  urlError <- tryCatch({httr::http_error(httr::GET(paste0(url, '?f=json'))},
                        error = function(cond) {TRUE})

Results against my server showing status code 200

> httr::status_code(httr::GET(paste0(paste(myserver, 'arcgis/rest/services', folderName, serviceName, 'FeatureServer', '1', sep="/"), '?f=json')))
[1] 200
> httr::status_code(httr::GET(paste0(paste(myserver, 'arcgis/rest/services', folderName, serviceName, 'FeatureServer', sep="/"), '?f=json')))
[1] 200
> httr::status_code(httr::GET(paste0(paste(myserver, 'arcgis/rest/services', folderName, serviceName, sep="/"), '?f=json')))
[1] 200
> httr::status_code(httr::GET(paste0(paste(myserver, 'arcgis/rest/services', folderName, sep="/"), '?f=json')))
[1] 200
> httr::status_code(httr::GET(paste0(paste(myserver, 'arcgis/rest/services', sep="/"), '?f=json')))
[1] 200
> httr::status_code(httr::GET(paste0(paste(myserver, 'arcgis/rest', sep="/"), '?f=json')))
[1] 200
> httr::status_code(httr::GET(paste0(paste(myserver, 'arcgis', sep="/"), '?f=json')))
[1] 200
> httr::status_code(httr::GET(paste0(myserver, '?f=json')))
[1] 200

I have not checked this solution against Map Services or ArcGIS online.

Note: I'm not an R programmer, today is day 1 with building R packages... I'm a python / GIS guy. We've been using esri2sf for the last couple years to report from our ESRI feature services (although I usually connect to them in Python) and recently it has broken, which led me to this solution. Our recent security hardening has caused us some pain in solving some issues like this one.

Thank you very much for developing this package. I successfully used esri2sf in November 2018, but since time has passed (including updating to R 3.5x) I'm getting an error.

running your example:

library("esri2sf") url <- "https://services.arcgis.com/V6ZHFr6zdgNZuVG0/arcgis/rest/services/Landscape_Trees/FeatureServer/0" df <- esri2sf(url) plot(df)

gives me the following output/error

[1] "Feature Layer" [1] "esriGeometryPoint" Error in names(value) <- nv : 'names' attribute [58] must be the same length as the vector [57]
The data frame is not created.

The same is true for my own dataset, where the "names" attribute is one longer than the length of the vector.

I attempted to install and run with R 3.4.3 with the same results, from RStudio Cloud to ensure the issue wasn't related to dependencies on my own machine.

I'm getting an error about invalid chars in json text. I think it's related to the number of results being returned, but not sure.

If I pass the additional argument resultRecordCount = 1045 the error does not occur (1045 seems to be the max value that will work, but it doesn't return all rows I expect).

I've put a reprex below, thanks.

bbox <- sf::st_bbox(c("xmin" = 148.724638, "ymin" = -35.509575, "xmax" = 149.424995, "ymax" = -35.123524), crs = 4326L)

url  <-  "https://geo.abs.gov.au/arcgis/rest/services/ASGS2016/SEARCH/MapServer/12"


# Fails
esri2sf::esri2sf(url, bbox = bbox)
#> Layer Type: Feature Layer
#> Geometry Type: esriGeometryPolygon
#> Service Coordinate Reference System: 3857
#> No encoding supplied: defaulting to UTF-8.
#> Error: lexical error: invalid char in json text.
#>                                        <!DOCTYPE html> <html lang=en> 
#>                      (right here) ------^

# Works
esri2sf::esri2sf(url, bbox = bbox, resultRecordCount = 1045)
#> Layer Type: Feature Layer
#> Geometry Type: esriGeometryPolygon
#> Service Coordinate Reference System: 3857
#> Output Coordinate Reference System: 4326
#> Simple feature collection with 1045 features and 6 fields
#> Geometry type: MULTIPOLYGON
#> Dimension:     XY
#> Bounding box:  xmin: 148.1424 ymin: -36.00292 xmax: 149.7606 ymax: -34.91801
#> Geodetic CRS:  WGS 84
#> First 10 features:
#>    OBJECTID SA1_MAINCODE_2016 Shape_Length Shape_Area SA1_7DIGIT_2016
#> 1        41       80107107112     3599.253  376559.82         8107112
#> 2       120       80101100311     1404.023   99378.99         8100311
#> 3       128       80109110108     2906.142  400432.40         8110108
#> 4       132       80107107104     1560.334  136348.11         8107104
#> 5       203       80104103511     1472.241  106309.49         8103511
#> 6       325       80106112902     3028.459  535893.94         8112902
#> 7       346       80108109707     2822.745  250644.43         8109707
#> 8       357       80107107905     2589.095  385617.58         8107905
#> 9       469       80107108220     2589.859  283930.20         8108220
#> 10      490       80101100206     1847.252  171195.43         8100206
#>    STATE_CODE_2016                          geoms
#> 1                8 MULTIPOLYGON (((149.1099 -3...
#> 2                8 MULTIPOLYGON (((149.0963 -3...
#> 3                8 MULTIPOLYGON (((149.1077 -3...
#> 4                8 MULTIPOLYGON (((149.0967 -3...
#> 5                8 MULTIPOLYGON (((149.1398 -3...
#> 6                8 MULTIPOLYGON (((149.1367 -3...
#> 7                8 MULTIPOLYGON (((149.0604 -3...
#> 8                8 MULTIPOLYGON (((149.1195 -3...
#> 9                8 MULTIPOLYGON (((149.0472 -3...
#> 10               8 MULTIPOLYGON (((149.0734 -3...

sessionInfo()
#> R version 4.3.1 (2023-06-16)
#> Platform: aarch64-apple-darwin20 (64-bit)
#> Running under: macOS Ventura 13.5
#> 
#> Matrix products: default
#> BLAS:   /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRblas.0.dylib 
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.11.0
#> 
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#> 
#> time zone: Australia/Melbourne
#> tzcode source: internal
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> loaded via a namespace (and not attached):
#>  [1] styler_1.10.1      utf8_1.2.3         generics_0.1.3     class_7.3-22      
#>  [5] KernSmooth_2.23-21 RSQLite_2.3.1      digest_0.6.33      magrittr_2.0.3    
#>  [9] evaluate_0.21      grid_4.3.1         fastmap_1.1.1      blob_1.2.4        
#> [13] R.oo_1.25.0        R.cache_0.16.0     jsonlite_1.8.7     R.utils_2.12.2    
#> [17] e1071_1.7-13       DBI_1.1.3          httr_1.4.7         purrr_1.0.2       
#> [21] fansi_1.0.4        pbapply_1.7-2      cli_3.6.1          rlang_1.1.1       
#> [25] crayon_1.5.2       units_0.8-3        R.methodsS3_1.8.2  bit64_4.0.5       
#> [29] reprex_2.0.2       withr_2.5.0        cachem_1.0.8       yaml_2.3.7        
#> [33] tools_4.3.1        esri2sf_0.1.1      parallel_4.3.1     memoise_2.0.1     
#> [37] dplyr_1.1.2        curl_5.0.2         vctrs_0.6.3        R6_2.5.1          
#> [41] proxy_0.4-27       lifecycle_1.0.3    classInt_0.4-9     fs_1.6.3          
#> [45] bit_4.0.5          pkgconfig_2.0.3    pillar_1.9.0       glue_1.6.2        
#> [49] Rcpp_1.0.11        sf_1.0-14          xfun_0.40          tibble_3.2.1      
#> [53] tidyselect_1.2.0   knitr_1.43         htmltools_0.5.6    rmarkdown_2.24    
#> [57] compiler_4.3.1

^{Created on 2023-08-21 with reprex v2.0.2}

I am not sure what is going on, but I get this:
ERROR: dependency ‘RSQLite’ is not available for package ‘esri2sf’

• removing ‘/Library/Frameworks/R.framework/Versions/4.0/Resources/library/esri2sf’
  Warning messages:
  1: In i.p(...) :
  installation of package ‘RSQLite’ had non-zero exit status
  2: In i.p(...) :
  installation of package ‘/var/folders/89/dr01tdl101lbqw42ly79rj300000gn/T//Rtmp4Ompdy/filef5db3bc922bc/esri2sf_0.1.1.tar.gz’ had non-zero exit status

and then

library("esri2sf")
Error in library("esri2sf") : there is no package called ‘esri2sf’

IMHO, the user should be able to change the CRS when needed. So far, esri2sfGeom() does not allow the user to do so.

I introduced this argument in my branch (https://github.com/inSileco/esri2sf) and fix some other issues (including #18) to have a clean build (meaning no warnings and no errors on devtools::check()), I can create a PR if you are interested.

This code is using the admin location to grab a Token. ERSI ArcGIS Online and other services use the 'public' url to get a token. On a secured server, the admin token url may not be available, but I think the 'public' one always is. The public one is at: "arcgis/tokens/generateToken" instead of "arcgis/admin/generateToken" as per: https://developers.arcgis.com/rest/services-reference/enterprise/generate-token.htm

Also, we can get iformation from the server info url, even on a server where directory services is disabled:
https://developers.arcgis.com/rest/services-reference/enterprise/server-info.htm

https://my.host.com/instance_name/rest/info?f=pjson

A json return looks like:

{
currentVersion: 10.61,
fullVersion: "10.6.1",
soapUrl: "https://geo.stantec.com/arcgis/services",
secureSoapUrl: null,
authInfo: {
    isTokenBasedSecurity: true,
    tokenServicesUrl: "https://geo.stantec.com/arcgis/tokens/",
    shortLivedTokenValidity: 120
    }
}

This info will be good for checking the intial validity.