Add Canadian DEM data about joerd HOT 8 OPEN

(Transcoding is complete and footprints are inserted)

from joerd.

Download link: http://ftp.geogratis.gc.ca/pub/nrcan_rncan/elevation/cdem_mnec/

from joerd.

This'll be a fun one to ingest because a) there are lots of subdirectories and b) everything is zipped.

Convenient commands:

# generate listings
wget -r --no-remove-listing --spider ftp://ftp.geogratis.gc.ca/pub/nrcan_rncan/elevation/cdem_mnec/

# read into a zip
gdalinfo /vsizip/cdem_dem_002C_tif.zip/cdem_dem_002C.tif

from joerd.

wget stopped being able to connect (probably because it was repeatedly reconnecting). None of the other ftp clients I tried first could do recursive listings.

However, lftp to the rescue!

cdec.lftp:

open ftp.geogratis.gc.ca
cd /pub/nrcan_rncan/elevation/cdem_mnec
find
exit

lftp -f cdec.lftp > cdec.list

Result: https://gist.github.com/mojodna/c807e3fdbe618e99681aaef8e326f5e6

from joerd.

Some images are only 1 or 4px high, which triggered some problems in footprint generation (root cause: rasterio/rasterio#1025)

Transcoded versions in s3://mapzen-dynamic-tiler-test/cdem/0/

from joerd.

(I'm basically repeating what @mojodna said above but more explicitly. These were all run from an instance in AWS.)

Listing files

lftp \
    -c 'find' \
    ftp://ftp.geogratis.gc.ca/pub/nrcan_rncan/elevation/cdem_mnec/ \
| grep tif.zip > nrcan_cdem_listing.txt

Mirroring files to openterrain S3

cat nrcan_cdem_listing.txt \
    | xargs \
      -I {} \
      -P 24 \
      -n 1 \
      sh -c 'export f="{}"; export s3="s3://elevation-sources-prod/nrcan_cdem/`basename $f`"; curl -s $f | AWS_DEFAULT_PROFILE=openterrain aws s3 cp - $s3; echo $s3' \
    tee nrcan_cdem_s3_objects.txt

Transcoding files

I got the nrcan_cdem_s3_objects.txt file onto my local computer, where I have the setup to submit transcoding jobs and ran:

cat nrcan_cdem_s3_objects.txt | \
while read filename; do
    bn=$(basename $filename)
    make submit-job job=aws/sample-transcode-job.json.hbs input=${filename} output=s3://elevation-sources-transcoded/nrcan_cdem/$(sed 's/_tif\.zip//' <<< $bn) name=$(sed 's/\./_/' <<< $bn)
done

from joerd.

It seems like we downloaded an older version of this data. There's a newer, higher-resolution dataset at http://open.canada.ca/data/en/dataset/957782bf-847c-4644-a757-e383c0057995?

Download link for this dataset is http://ftp.maps.canada.ca/pub/elevation/dem_mne/highresolution_hauteresolution/dsm_mns/2m/

from joerd.

Let's track that newer data in a different issue: #152

from joerd.