fieldhand / fieldhand Goto Github PK

Currently, the Paginator is responsible for creating an HTTP client which actually makes the necessary requests to the OAI-PMH endpoint but offers no way for the user to set an open or read timeout for that client.

We should likely set some default (so that connections don't hang indefinitely) and allow users to configure their own timeouts.

During harvesting the service randomly returned 500 server error

Fieldhand::ResponseError (Invalid response: 500 Internal Server Error)

Due to this I had to start harvesting from the beginning, because there is no mechanism to retry. It was really annoying on large dataset, so I end up monkey patching Fieldhand::Paginator#request method.

module Fieldhand
  class Paginator

    private

    alias_method :original_request_method, :request

    def request(query = {})
      send(:original_request_method, query).tap do |response|
        logger.debug('Fieldhand') { response.body }
      end
    rescue => e
      retries = retries.to_i + 1
      if retries < 30
        delay = retries * retries
        logger.warn('Fieldhand') { "Error fetching response: [#{e.class}] #{e}\nRetrying in #{delay} seconds" }
        sleep delay
        retry
      else
        raise e
      end
    end
  end
end

It would be really helpful to have such a mechanism in a gem.