they say sckott for github user/org right now

http://www.gbif.org/developer/registry#oai-pmh

Hi,

many thanks for this package; it looks just like what I'm after.

Can someone help me get started? I'm trying to use oai to interact with DSPACE repositories, but neither of the two that I tried worked. What am I missing?

> id("https://dash.harvard.edu/oai")
Error in read_xml.raw(charToRaw(enc2utf8(x)), "UTF-8", ..., as_html = as_html,  : 
  Opening and ending tag mismatch: meta line 6 and head [76]

and

> id("https://www.repository.cam.ac.uk/oai")
Error in read_xml.raw(charToRaw(enc2utf8(x)), "UTF-8", ..., as_html = as_html,  : 
  Opening and ending tag mismatch: meta line 6 and head [76]

Thanks, Stephen

To get the development version of DBI so you can run the checks
yourself, you can run:

# install.packages("devtools")
devtools::install_github("rstats-db/DBI")

To see what's changed, visit
https://github.com/rstats-db/DBI/blob/master/NEWS.md.

trunc_mat and other functions in tbl_df.R could use at least a one line commented description

Looked at times of tests, seems there's two we need to speed up. I'll work on list_identifiers. @mbojan - can you make the dumpers tests faster?

                           .id user.self sys.self elapsed user.child sys.child
10            test-providers.R     0.028    0.001   0.042      0.000     0.000
8          test-list_records.R     0.088    0.002   0.606      0.000     0.000
3           test-get_records.R     0.092    0.003   0.787      0.000     0.000
7  test-list_metadataformats.R     0.100    0.004   0.992      0.000     0.000
5                    test-id.R     0.090    0.004   2.443      0.000     0.000
4         test-handle_errors.R     0.079    0.010   3.266      0.000     0.000
1     test-count_identifiers.R     0.265    0.020   6.917      7.717     0.097
9             test-list_sets.R     1.012    0.018   7.045      0.000     0.000
6      test-list_identifiers.R    19.338    0.181 111.121      0.000     0.000
2               test-dumpers.R    19.518    0.680 259.208      0.000     0.000

Try:

out <- oai::list_records(url="http://export.arxiv.org/oai2",
  prefix="arXiv",
  from='2022-04-01',
  )

This results in:

Service Unavailable (HTTP 503)

No results are returned, even though partial results were collected. So there is no graceful way to resume.

Their are three issues here that seem to make the overall interface non-robust

1. there is no mechanism for setting a delay between subsequent requests ... so
2. the arxiv server eventually issues a 503 flow control directive -- which the client treats as a permanent failure -- rather than
   an instruction to delay for the specified value and retry. Causing the client to stop() which ...
3. aborts the call, rather returning partial results ... so no resumption is possible.

A possible workaround could be to write an external wrapper that divides up the "from" - "to" interval into small chunks, and uses purr:: wrappers to schedule each chunk and retry... This is inelegant. A cleaner solution might be to handle the OAI flow control explicitly internally in while_oai(), and to at least return partial values and a resumption token on error.

Running count_identifiers with more than one URL triggers a warning because is.url gets a vector and returns a vector, which is later used in if(), and if requires a scalar.

This also happens in one of the tests.

Probably would be enough to use all() somewhere.

https://www.openarchives.org/OAI/openarchivesprotocol.html#ErrorConditions

I think currently only HTTP errors are handled through httr::stop_for_status in while_oai.

Extracted from #17.

Internet-related errors, like timeouts etc. They are thrown by httr::GET, which is used in several places, but most importantly in while_oai, which is the core of list_* functions.

• Things like timeouts could be addressed by proper pre-configuring httr::GET (passing additional arguments through httr::config
• Perhaps it should be possible to add an optional sleep time between requests in while_oai.

We are using zenodo.org as a oai compatibe repository.
Zenodo encodes certain "subjects" in this format in the xml:

<subject subjectScheme="url">http://id.agrisemantics.org/gacs/C8154</subject>

This type of subjetcs do not appear in the data.frame, if I call list_records, like this:

get_records(c("oai:zenodo.org:159890"),url="https://zenodo.org/oai2d",prefix="oai_datacite")

I need to choose the "oai_datacite" prefix, if not the server does not return the records at all.

So by using prefix="oai_datacite" I can see that they get returned by using the "raw" option, like this:

get_records(c("oai:zenodo.org:159890"),url="https://zenodo.org/oai2d",prefix="oai_datacite",as="raw")

But the the later parsing omits them somehow. I looked at the code and debuged it, but could not find where excactly they get lost.

In case users want that

OAIHarvest returns XML or array output, seems good to return not just data.frames

@mbojan when you get a chance, could you submit a new version to cran changing the maintainer to you? not urgent, but any correspondence from cran will go to me right now, so it'd be good to have it go to you instead. thanks!

Travis trips with

test_check("oai")
── 1. Error: list_identifiers - set (@test-list_identifiers.R#31)  ─────────────
OAI-PMH errors: noRecordsMatch: The combination of the values of the from, until, set, and metadataPrefix arguments results in an empty list.
1: list_identifiers(from = "2011-06-01T", until = "2012-11-01T", set = "CDL.OSU") at testthat/test-list_identifiers.R:31
2: while_oai(url, args, token, as, ...)
3: handle_errors(parsed)

See the request made by

list_records(token="foo", config=httr::verbose())

The request should contain the parameter resumptionToken (not token) and no other parameters (without metadataPrefix). So it should be:

r <- httr::GET("http://oai.datacite.org/oai?verb=ListRecords&resumptionToken=foo",
     config=httr::verbose())
httr::content(r, "text")   

which properly includes badResumptionToken error code for resumptionToken=foo

Perhaps also list_records when given a token should compose the URL without any other arguments whatever user supplied or give errors immediately, for example:

list_records(from="2015-09-01T", until="2015-09-01T", 
    token="foo", config=httr::verbose())

should skip not only the prefix (which puts some default), but also from and until and so on

as attribute, perhaps

Extracted from #17

Malformed XML errors - it may happen that the downloaded XML is malformed and cannot be parsed by xml2::read_xml. For example, contains UTF-8 characters that are illegal for XML, that are not escaped etc.

If the XML is malformed (3 in #17) it could be written somewhere without parsing. That would require some fallback mechanism of getting the resumptionToken without parsing the whole XML with read_xml. One option is just to write a regular expression. Perhaps another option is related to r-lib/xml2#10 if it gets implemented.

Hi Scott,
the last commit added a call for check_as

Maybe something like this?

# check as ----------------------------------------
check_as <- function(as) {
  if (!as %in% c('parsed', 'raw')) {
    stop(sprintf("'%s' not in acceptable set: parsed, raw", as) ,
         call. = FALSE)
  }
}

👋 Patrick

been so for a while now. b/c won't install Rtools if no /src dir, but seems to work fine for some pkgs with no /src dir, weird.

descirbe more detail, give some links, some examples

Noticed something while making fix in #18 - that we probably need to give user power to determine how much data they get back.

Our while loop will just keep going, getting more data if a resumptionToken is available.

We may not be able to expose a parameter that does this exactly, like limit = 10, and you get 10 results, but at least it could be something like

• limit = "all" (all results)
• limit = "one" (do one HTTP request, then stop, so even if get a resumptionToken, stop anyway)
• other options?

@mbojan I can't figure out why they are not working. haven't thought about it too hard yet, but I imagine you can figure it out faster.

ive commented those tests out for now.

Hi @sckott , I have the following question: would you have anything against including the information about the package authors in the oai-package.R file? That would allow me to insert there the grant citation that was supporting my contributions to this package. I imagine a section "Authors" with a bullet for each author and mine as something like "Michal Bojanowski, supported by NCN grant 2012/07/D/HS6/01971". This could be also in README. Or perhaps you'll have a different suggestion.

Bit more background: I used the oai package to collect data for research purposes for my project http://recon.icm.edu.pl . Now I am in for writing the final report and listing various contributions made. I wanted to include my work on extending oai (dumpers etc.) for data collection purposes. This will have no copyright/licensing consequences whatsoever.

What do you think?

right now, I think function calls error when as does not equal one of the accepted values, raw, df, list

This is a rather general/thick issue that will have to be split.

At this moment harvesting functions do not have any recovery protocols. This is rather painful with larger requests to OAI-PMH because there is no good way of splitting a request into chunks apart from resumptionToken.

In general there are the following types of errors:

1. OAI-PMH errors - currently handled by handle_errors
2. Internet-related errors, like timeouts etc. They are thrown by httr::GET, which is used in several places, but most importantly in while_oai, which is the core of list_* functions.
3. Malformed XML errors - it may happen that the downloaded XML is malformed and cannot be parsed by xml2::read_xml. For example, contains UTF-8 characters that are illegal for XML, that are not escaped etc.

It would be useful to come up with a way to recover from such errors. For example:

• If the XML is malformed (3) it could be written somewhere without parsing. That would require some fallback mechanism of getting the resumptionToken without parsing the whole XML with read_xml. One option is just to write a regular expression. Perhaps another option is related to r-lib/xml2#10 if it gets implemented.
• Timeouts could be addressed by proper pre-configuring GET, but also by adding an optional sleep time between requests in while_oai.
• resumptionToken usually comes with a expirationDate so in order no to overload the server, harvesting could wait some time, not longer that the expirationDate, until issuing next request with the token.
• ...

It would be nice if you could read the local xml files that were downloaded by using dump_raw_to_txt or another harvester. I imagine a function list_local_records(startfile = "/path/to/file", prefix = "oai_dc"), which reads all the local dump files using the resumptionToken in the startfile. Similar to list_records but for local files.

Is there a reason why while_oai and other functions use nested xml_children calls to extract various elements from the result (like resumptionToken, errors, etc.)? I think the OAI-PMH specification does not specify the order of the tags so different services might return these in different order. Using XPath seems simpler.

One trouble with XPath is handling namespaces. This might be dodged with local-name() like here http://stackoverflow.com/questions/16717211/getting-elements-with-default-namespace-no-namespace-prefix-using-xpath

Currently results returned by list_* functions can be "raw" XML, XML past to a "list" or to "df". It could rather easily return additionally:

• as="xml" - parsed XML as a xml_document object (in while_oai use xml_orig)
• as="xml_list" - a verb-related subtree of XML, e.g. for list_records the tag <ListRecords> (in while_oai use xml object).

XML is already parsed so we get those for free. It is useful to use them in dumper functions - no need to parse raw XML again.

from @mbojan

the test for list_identifiers throws an error when there are no results. This just happened for me when running R CMD check - there where no data yet for today. For now I modified the test to look up updates from yesterday.

Write a oai_available(url) that checks if there is a working OAI-PMH service on url.

• Executing some of the tests might be conditional upon that.
• It may also be useful to the end-user.

Inspired by "Skipping a test" here http://r-pkgs.had.co.nz/tests.html . Mentioned in #24

I'm sorry to ask these questions, since they might have obvious answers, but I couldn't find anyone's code using this library (which is how I normally learn how to use a package). So here it goes:

I'm trying to parse this the Dublin Core from this OAI-PMH Request URL I would like to have it as tibble (which includes the results from of the resumptionTokens) is there a way to do this with this client?

Thanks for this pkg!

I need to harvest and process rather large requests and merging all results of a single request into one R object is a nono (not enough RAM). For that purpose a forked (https://github.com/mbojan/oai) and modified while_oai to call some dumper function that might e.g. write results to a DB page-by-page (resumptionToken).

Perhaps you are interested in such an enhancement? Suggestions welcome. I'm still working on it.

Many of the other methods allow callers to specify a format, but id() doesn't. This would be very useful to access the extended XML defined by many software packages.