Deals with compression (if necessary)

append?

I think query_item_identifier is supplying a new, incorrect session when the session arg is not explicitly set. This may be because this test passes if session is undeclared in the args, even though the default to session is current_session():

if (missing(session) || is.null(session)) {
    session = handle(pkg.env$url_base)
}

So there might be two problems. 1, the session gets reset to handle(pkg.env$url_base) even when you think you're choosing the default arg value. 2, the value that session gets set to is different from the value returned by current_session(). See:

> httr::handle(sbtools:::pkg.env$url_base)
# Host: https://www.sciencebase.gov/catalog/ <0x0000000011f31b10>
> current_session()
# Host: https://www.sciencebase.gov/catalog/ <0x0000000011cdbc00>

Not sure why that different session w/ same URL is tripping things up, but this turns into a problem for me in continental stream metabolism, where I created this item:

# find the project root ("Continental Stream Metabolism" folder)
true_sites_root <- sbtools::query_item_identifier(scheme="mda_streams", type="project_root", key="uber")$id
project_root <- sbtools::item_get_parent(true_sites_root)

# create a sandbox sites folder to work with from here on
sites_root <- sbtools::item_create(parent_id = project_root, title="Sites_dev")
sbtools::item_update_identifier(id=sites_root, scheme="mda_streams_dev", type="sites_root", key="uber") # true sites root currently has type="project_root". i find that confusing.
sites_root_saved <- sites_root
> sites_root_saved
# [1] "55568a6fe4b0a92fa7e9cf2d"

then i tried to find it again, but got an empty data.frame:

sites_root <- sbtools::query_item_identifier(scheme="mda_streams_dev", type="sites_root", key="uber")
> sites_root
# data frame with 0 columns and 0 rows

but if i explicitly declare session=current_session(), it works.

sites_root <- sbtools::query_item_identifier(scheme="mda_streams_dev", type="sites_root", key="uber", session=current_session())
> sites_root
#       title                       id
#1 Sites_dev 55568a6fe4b0a92fa7e9cf2d

instead of NULL or df with nrow = 0.

when

item_list_files(data.frame(), session = NULL)

tried this with item_append_file(), but it supports an additional file with the exact same name (does not overwrite).

prompt shows up anyway. Should fail before that.

Does this fit in sbtools?

quick way to do it is this:

get_parent = function(item_id){

  url <- sprintf("https://www.sciencebase.gov/catalog/item/%s?format=json&fields=parentId", item_id)

  parent_id <- fromJSON(txt = url)$parentId

  url <- sprintf("https://www.sciencebase.gov/catalog/item/%s?format=json&fields=title", parent_id)
  parent_site <- fromJSON(txt = url)$title
  return(parent_site)
}

but we would ask for item json twice per id. Any better way to do this?

From httr::content:
When using content() in a package, DO NOT use on as = "parsed". Instead, check the mime-type is what you expect, and then parse yourself. This is safer, as you will fail informatively if the API changes, and you will protect yourself against changes to httr.

See r-lib/httr#246

Currently, this is speckled throughout the package.

should be d.f w/ zero rows per design pattern

item_list_files(id = '54876132e4b02acb4f0c87da', session)
  fname size url
1    NA   NA  NA

authenticate_sb

saw mention of ncdf in the proposal and had a comment (and I don't work with ncdf formats much, so I'm kinda clueless here):

I hear from Roy Mendelssohn at NOAA that the ncdf package http://cran.r-project.org/web/packages/ncdf/index.html only works with older ncdf format. True? But this pkg is nice b/c it installs on all OS'es.

Roy said to instead use ncdf4 http://cran.r-project.org/web/packages/ncdf4/index.html - but there are no windows binaries, and it sounds as though there never will be.

Thoughts on this? I ask b/c if we need ncdf4 functionality, that could lead to a problem for windows users that aren't super savvy (aka that couldn't do install from source, etc.)

Use case is wanting to find public data based on spatial and/or keywords and type etc.

for public items.

right now it creates the item with the same name as the file.

should be added? works on item_list_files but w/ a GET()

Error in parseJSON(txt) : parse error: premature EOF {"total":1,"selflink":{"rel":"s (right here) ------^

Says it returns a NULL when no matching item found:

query_item_identifier(scheme = 'mda_streams', type= NULL, key = sites, session, limit = 10000)
data frame with 0 columns and 0 rows

would default to using names and dest_dir

see DOI-USGS/mda.streams#32

from @dblodgett-usgs

would be nice if you download files function returned the paths to the files or at least the file names it downloads instead of true, but that’s minor.

We could return a list of all files downloaded with their paths, regardless of if the user supplied file names or if names were generated. This would be more useful than TRUE.

Might take a look at what we are doing w/ hazardItems auth

After auth works, we set pkg.env token and username, and do a token check when session is needed.

you get a 404 if the item has >0 files associated with it. Currently checks for children but not files

internal tracking of session stale should be re-upped here.

The motivating example is timeout(), which users may occasionally need to specify.

• curl options - I'd suggest using ... or similar to allow curl options to be passed
  into GET/POST/etc.
• authentication: I haven't dug into this, but i imagine passing in the
  session bit to each function call makes sense more so if there are multiple
  potential accounts a user could have, while if a user will only ever have
  one, then perhaps the auth function could be run once by a user, and then
  run internally within function calls so the user doesn't have to worry about
  it

looking through this more, but these are the first two things...

when using multi-file post

seems to break the pattern of being able to pass around sessions.

Thoughts @lawinslow ?

not the upsert I'm after in #37 , but simpler to implement and useful for other things. @lawinslow thoughts on whether this is worth adding?

make this make more sense to people that their session is out of date:

Error in RCurl::curlPerform(curl = handle$handle, .opts = curl_opts$values) : 
  Stale CURL handle being passed to libcurl 

reverse of item_list_children()

See this example:
https://www.sciencebase.gov/catalog/item/editForm/54890a0be4b0772072e5a653

JSON parsing BS

@mhines-usgs any interest in helping out with this?

as I'm adding dots arguments today to pass to curl, this is the first case i've seen where the same dots have to be passed to multiple functions - in this case, within item_update_identifier, the dots go to both query_item_identifier and to sbtools_PUT. I'm introducing this oddity because I don't know how else to handle the dots - should we be accepting two different well-formed, single-item config lists rather than allowing the user to pass in the info in standard curl dots format?

currently fails when name is missing

item_file_download fails cryptically, as does item_list_files

We need to adjust the language of the disclaimer on GRAN to:

.onAttach <- function(libname, pkgname) {
  packageStartupMessage("This information is preliminary or provisional and is subject to revision. It is being provided to meet the need for timely best science. The information has not received final approval by the U.S. Geological Survey (USGS) and is provided on the condition that neither the USGS nor the U.S. Government shall be held liable for any damages resulting from the authorized or unauthorized use of the information.")
}

for example, item_upload_create() will return NULL if you aren't authorized to perform that action

library(sbtools)

myFolder <- "550057b9e4b02419550fa5f7"
session <- authenticate_sb(username = "xxx")

folderID <- item_create(myFolder, 
                        title="Test Workflow",
                        session=session)

fileStuff <- item_append_files(folderID,
                               files = "fluxBiasMulti.pdf",
                               session = session
)

x <- item_update_identifier(folderID, 'test', 'workflow', "Unique thing", session )
# This wipes out the identifier:
fileStuff <- item_append_files(folderID, files = "multiPlotDataOverview.pdf", session = session)

I can add:

x <- item_update_identifier(folderID, 'test', 'workflow', "Unique thing", session )

after each item_append_files...but sometimes I get an error that there already is an identifier...presumably adding in a lag might prevent that.

local timeout of 55 minutes, as a pkg.env var