CKAN: Datasets can be missed due to paging about magda HOT 5 CLOSED

Have we considered doing any smarts on the incremental package_search, using metadata_modified to order - and page until we meet a record already updated? Whilst this may not solve up the issue, it will minimise the time it is missing, and minimise the chance of it occurring because of the reduced time querying?

This should enable more frequent increments (say hourly), coupled with a daily full refresh?

from magda.

yeah that's a good idea for doing incremental updates. I believe CKAN also has an API for getting modified packages since a date. But any kind of incremental update requires us to be able to do a full crawl and know that we got everything that was last modified before a certain date (the beginning or end of the crawl, let's say, depending on how the initial crawl works). So this issue is about making sure the initial crawl is actually reliable.

from magda.

Although maybe what you're suggesting is that we make a full crawl reliable by doing it naively, and then following it up with incremental one(s) until we don't see any more changes. That could work, and might be simpler than the page-with-overlap thing I described.

from magda.

yeah - if a full crawl is always followed by an incremental it should fix any updates. The same logic may also be applied to "large" multi-page incrementals for surety?

from magda.

Fixed by 6a546f6 using the simple approach of asking package_search to sort results by metadata_created asc. The order shouldn't be affected by updates, and new datasets should appear at the end.

from magda.