codeforiati / iati-data-dump Goto Github PK

Describe the bug

The iati-data-main.zip file contains datasets that are no longer available.

Here is a prime example:
FCDO Zimbabwe is currently unavailable
However, if you download the latest iati-data-main.zip, the original .xml file with its data is available there.
It does show up in the error log as not available.

This means that the data still enters our processing pipeline while not actually available.

Oddly enough the validation status on the registry has not changed from "error" to "not found" registry fcdo zw link, and the metadata seemingly has not updated to reflect the new file content.

But it does not seem like it should be possible for the data dump to contain the old files (what if a publisher needs to redact its files like we saw with the Afghanistan situation).

Hopefully we can quickly resolve this! Happy to chat about it.

There's been no new data on https://iati-data-dump.codeforiati.org/ since Saturday.

This is because you've run out of space on GitLab.

e.g. https://github.com/codeforIATI/iati-data-dump/runs/7969924999?check_suite_focus=true

remote: GitLab: Your push has been rejected, because this repository has exceeded its size limit of 10 GB by 11 GB. Please contact your GitLab administrator for more information.

wget appears to get stuck while trying to download data from ausgov. Changing timeouts doesn’t appear to help.

I’ve just tried switching to curl, which might help.

@shi shrewdly observes that this would ideally be rsyncable.

I’m a bit doubtful I’ll do it, just because I’m not sure it’s possible using exclusively free or freemium services. But that’s a shame, because this is a really great idea.

Describe the bug

Every 60 days, the scheduled github action gets switched off by github due to inactivity on the repo.

Expected behaviour

The scheduled action should just keep running.

Screenshots

Additional context

The action should auto-push a commit to a branch, just to keep the repo active.

Some tips and examples here:
https://help.github.com/en/articles/setting-guidelines-for-repository-contributors