Serve compressed CSV about machine HOT 5 CLOSED

If we gzip the CSV before we put it on S3 and then tell S3 that it's
gzipped they will serve it with the appropriate headers that trigger a
browser to decompress it before showing/downloading it to the user.

from machine.

Unfortunately always serving gzip breaks HTTP; you're not supposed to send it gzip unless the client asks. It only works if the browser expects and supports Content-Encoding: gzip. curl and wget both explicitly don't deal with gzip by default.

I set up a demo of this at http://com.somebits.s3gzip.s3-website-us-west-2.amazonaws.com/gzipped.txt. It loads fine in Chrome. It fails with curl but works with curl --compressed. wget doesn't even have a way to decompress, you have to do wget | gunzip. That's why I think it's better to explicitly name the file .csv.gz; the user will know what to do with it.

Lots more detail about this weirdness in S3 on my blog.

from machine.

@migurski mentioned running out of disk in some of his EC2 runs. If we only produce compressed products then we can save a lot of disk while a job is running. A full run of 790 sources wrote 28G to disk. If I tar + gzip the whole directory that gets to 17G, 60%. Not an enormous savings but it can only help. (I suspect it's not more because a lot of the 28G is cached shapefile zips that are already compressed.)

from machine.

We had a discussion in Slack and @migurski, @iandees, and I all agree we should zip the CSV output and serve .zip files instead of .csv files. This lets us avoid the whole S3/gzip encoding question. Also allows us to put extra files into the product if we want, maybe a .vrt file or some other metadata.

from machine.

PR #84 does the thing—one of you want to have a look before I merge?

from machine.