Handling XML errors about oai HOT 14 CLOSED

I'm on it.

from oai.

Yet another option for getting the resumptionToken is to use xml2::read_html. It seems to be much more tolerant with malformed files (including illegal characters which seem to be deleted).

It should not be used instead of read_xml though, because, among other things, it makes or XML tags in lower case. But seems OK if we are only interested in getting the token.

from oai.

I am trying the following modification for while_oai to (try to) proceed with harvesting even in case of XML faults:

while is.character(token)
  GET()
  try to parse XML with `read_xml` (with optional removal of invalid characters)
  IF parsing is ok
    check for oai-pmh errors
    look for `resumptionToken`
    process the results as determined by `as` and `verb`
    collect the results in `out` and/or pass to dumper
  ELSE (i.e. read_xml fails)
   try to parse with `read_html`, if fails dump raw to file and stop()
   check for oai-pmh errors
   look for `resumptionToken`
   IF `as="raw"` 
     collect raw results in `out` and/or pass to dumper
   ELSE
    dump raw XML to a file
    warning("bad XML dumped to file")
  IF has `resumptionToken`
   token <- resumptionToken
  ELSE
   token <- 1

from oai.

The above assumes that the result of parsing with read_html is unreliable. So we write raw XML to a file and try to proceed with the resumptionToken if any.

from oai.

@mbojan tests are now failing on handle_errors fxn, the class returned is no longer oai-pmh_error, but Rcpp::exception - any thoughts?

run the test suite to see what happens

from oai.

Looks like OAI-PMH service at pbn.nauka.gov.pl is malfunctioning (certificate problems). Only those tests seem to fail.

from oai.

Hmm, okay, anything we should do to fail better in those cases?

from oai.

I'll change the test URLs and see if it the tests pass correctly.

What was failing is actually httr::GET not oai error handling. These test are suppose to test the correct catching of OAI-PMH errors conditional upon an assumption that the test URLs actually lead to these errors. So I don't think there is a need to modify the tests apart from coming up with URLs that are correctly returning OAI-PMH exceptions from a fully functional OAI-PMH server. What do you think?

That's a general problem with testing your system against some external system...

from oai.

Okay, i'll have a look at the http request error catching

from oai.

Do you think they deserve a dedicated "net" of tests to catch?

One thing I might add is the OAI-PMH error handling tests first check whether the request returns a proper result at all before parsing it to learn what the OAI-PMH exception is.

from oai.

Do you think they deserve a dedicated "net" of tests to catch?

we'll see, I'll look into it

first check whether the requests a proper result at all before parsing it to learn what the OAI-PMH exception is.

makes sense

from oai.

Perhaps it makes sense for those tests that rely on contacting some OAI-PMH service to first check whether the service is available and then skip the tests if it is not available?

Inspired by "Skipping a test" here http://r-pkgs.had.co.nz/tests.html .

We would have to write something like oai_available(url) though.

from oai.

yeah, sounds good

from oai.

closing for now, we can open a new issue if ther'es still problems along these lines

from oai.