Global Historical Climate Network - Daily about datasets HOT 11 OPEN

running a mirror of the linked ftp site.

from datasets.

FileZilla says there is 5.1 TB of data in there

from datasets.

Thanks for the warning @as-com relaunched on a server with 9T of scratch space, though I'll need to find a better home in a few weeks. I suppose worst case I could buy a hard disk or two.

from datasets.

Overnight only transferred 13 GB, trying to increase number of parallel requests, though that's generating warnings.

Dear Scientists, please consider including bittorrent links for your public data.

from datasets.

Have ~690G so far

from datasets.

Is there a slack team? Could I join?

from datasets.

@nickrsan can you send @detrout an invite to slack? thanks!

from datasets.

From the README it looks like we probably don't need most of this 5 TB. In summary I think we need ghcnd-all.tar.gz and the metadata .txt files, which is only ~3 GB. I think most of the directories are multiple uncompressed copies of the data in ghcnd-all.tar.gz.

The grid directory is described as "Directory with the GHCN-Daily gridded dataset known as HadGHCND", so it couldn't hurt to grab it as well (~5 GB).

from datasets.

Actually, like the the main directory we just need ghcnd.grid.tar.gz and the README from the grid directory, so < 1GB. I now have copies of all everything I mentioned down.

from datasets.

Well I have all 4.7 T. Though I need to figure out how to update the daily portions of the mirror

from datasets.

I've been occasionally rerunning mirror to update the archive.

I just done a zfs snapshot so I can see easily see whats changed between mirrors.

from datasets.