mbennett-uoe / librarytools Goto Github PK

If the first result after a sleep is found in the cache, the query hit counter obviously won't be upgraded and so the rate limiter triggers again.

Possible fixes:

• Move position of sleep logic block.
• Add slept_last_time state handler and logic

Add a flag and some new logic

This is slowing us down quite a lot, because each duplicate object counts as a hit towards the rate limiter, even though no OCLC query is made, meaning that in a set of 500 records with 250 duplicates, the longer sleep will get triggered twice instead of once

This might be useful for stuff matched with non-exact matching, as it could be used to filter rows that got a large number of hits and checking that the retrieved data is accurate.

Use vprint function to expose data about search process etc

Maybe scaling based on response time?

Was triggering on first loop due to index being 0 and 0 mod 250 = 0. Adding +1 to index has stopped it triggering at all.

Maybe need to explicitly declare the changed index, i.e is it trying to eval "index + (1 % 250) == 0" instead of "(index+1) % 250 == 0"?

Maybe only on the longer sleep, but this would be a good insurance policy against a crash midway through processing a large file.

When -f flag is used, it would be nice to give the user the chance to change the fields or add ones the script couldn't find.

csv.DictWriter seems to be ignoring the order of fields in the list that is passed to it. List order is correct in the supplied argument.