Add 5 m data for Australia about joerd HOT 16 OPEN

Thanks!

from joerd.

Looks like there's a new 1m and 2m DEM for New South Wales, see #121.

from joerd.

Data download URLs:

• https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_NSW.zip
• https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_VIC.zip
• https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_QLD.zip
• https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_NT.zip
• https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_SA.zip
• https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_TAS.zip
• https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_WA.zip

from joerd.

Listing files

curl -s http://www.ga.gov.au/metadata-gateway/metadata/record/gcat_22be4b55-2465-4320-e053-10a3070a5236 \
  | grep -o '<a href=['"'"'"][^"'"'"']*['"'"'"]' \
  | sed -e 's/^<a href=["'"'"']//' -e 's/["'"'"']$//' \
  | grep state-mosaics \
  | sort \
  | uniq \
  > geoscience_au_listing.txt

Mirroring files

I skipped mirroring because these were already on S3 and are rather large.

cat geoscience_au_listing.txt \
    | xargs \
      -I {} \
      -P 24 \
      -n 1 \
      sh -c 'export f="{}"; export s3="s3://elevation-sources-prod/geoscience_au/`basename $f`"; curl -s $f | AWS_DEFAULT_PROFILE=openterrain aws s3 cp - $s3; echo $s3' \
    tee geoscience_au_s3_objects.txt

Transcoding files

cat geoscience_au_listing.txt | \
while read filename; do
    bn=$(basename ${filename%.*})
   make submit-job job=aws/transcode-job.json.hbs input=${filename} output=s3://elevation-sources-transcoded/geosciences_au/${bn} name=${bn:0:50}
done

from joerd.

These seem to be failing intermittently, probably because the ECS instance is running out of disk space. I'll check these off as they successfully run.

• NSW
• VIC
• QLD
• NT
• SA
• TAS
• WA

from joerd.

…these are still failing, even when running on a separate instance with plenty of space. It's not clear to me why they're failing, but it seems that gdal_translate is just stopping at some point:

Processing https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_NSW.zip to s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW...
Downloading https://s3-ap-southeast-2.amazonaws.com/elvis.ga.gov.au/elevation/5m-dem/state-mosaics/22be4b55-2465-4320-e053-10a3070a5236_NSW.zip...
Transcoding...
Transcoding zip:///tmp/tmp.DXfBGJtkdW.22be4b55-2465-4320-e053-10a3070a5236_NSW.zip!NSW/z54/mdbaz54ag.tif
Transcoding bands...
Input file size is 84000, 104800
0...10.delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW.tif
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW.tif.msk
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW_footprint.json
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW.vrt
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW_thumb.png
delete: s3://elevation-sources-transcoded/geosciences_au/22be4b55_NSW.json

from joerd.

That behavior is consistent with things I've seen when there hasn't been enough memory available. GDAL tries to be sensible about what it's using and seems to end up never finishing. This is why it's wrapped with timeout in places. Try increasing the amount of memory available?

from joerd.

@mojodna I'm doing this on a machine with 64GB of memory and don't see any memory pressure while I'm running it. I wonder if Docker limits memory usage by default?

from joerd.

I don't think it does, so maybe this is something else... Hmm.

from joerd.

After trying this again on a bare EC2 instance I tried downloading a DEM to my laptop. Unzipping the file that got downloaded with the MacOS X expander created a .cpgz file instead of the expected .tif, so I tried it from the command line and got:

$ unzip 22be4b55-2465-4320-e053-10a3070a5236_TAS.zip
Archive:  22be4b55-2465-4320-e053-10a3070a5236_TAS.zip
   skipping: TAS/tasz55.tif          need PK compat. v4.5 (can do v2.1)
  inflating: TAS/Ancillary/metadata/Digital Elevation Model (DEM) 5 Metre Grid of Australia derived from LiDAR.pdf

So... maybe it's just a broken zipfile?

from joerd.

This data is all over the place and missing most of Australia. Since there is no good, easily accessible file for Australia let's travel over to New Zealand where LINZ is straight forward.

from joerd.

Is this project still active? Is there any plan for getting better resolution data for Australia at present?

from joerd.

It's active, but the Earth doesn't change all that much so we don't update it frequently.

Some Australian data made it into the most recent build, but the data we found here didn't cover the entire country. I think there's just a few small pieces that we found. We'd love to see sources for better data in Australia!

from joerd.

As far as I can tell from http://elevation.fsdf.org.au

NSW provides state wide ranging from 1m-5m CC BY.
TAS is not under an open license.
VIC provides state wide 10m CC BY (http://www.delwp.vic.gov.au/vicmap) and coastal areas 0.5m-1m is not under an open license.
QLD not under an open license

I've been doing the clip and ship process for about 1/3rd of NSW, and have this sitting in a private Digital Ocean Space, which I can share if anyone is interested in bypassing the Clip and Ship process.

from joerd.

Does this not provide 5m coverage for all of Australia that is CC BY?
http://gaservices.ga.gov.au/site_9/rest/services/DEM_LiDAR_5m/MapServer

from joerd.

It's not easy to get bulk data out of an Esri REST map service. I think that service uses the data mentioned on this ticket above that doesn't successfully unzip.

I'm downloading the data from here: https://data.gov.au/dataset/ds-ga-22be4b55-2465-4320-e053-10a3070a5236/details / https://ecat.ga.gov.au/geonetwork/srv/eng/catalog.search#/metadata/89644 (which may or may not be an updated version of what I linked to above). Once I get it downloaded I'll see if the zip files are still broken.

from joerd.