Comments (8)
(Transcoding is complete and footprints are inserted)
from joerd.
Download link: http://ftp.geogratis.gc.ca/pub/nrcan_rncan/elevation/cdem_mnec/
from joerd.
This'll be a fun one to ingest because a) there are lots of subdirectories and b) everything is zipped.
Convenient commands:
# generate listings
wget -r --no-remove-listing --spider ftp://ftp.geogratis.gc.ca/pub/nrcan_rncan/elevation/cdem_mnec/
# read into a zip
gdalinfo /vsizip/cdem_dem_002C_tif.zip/cdem_dem_002C.tif
from joerd.
wget
stopped being able to connect (probably because it was repeatedly reconnecting). None of the other ftp clients I tried first could do recursive listings.
However, lftp
to the rescue!
cdec.lftp
:
open ftp.geogratis.gc.ca
cd /pub/nrcan_rncan/elevation/cdem_mnec
find
exit
lftp -f cdec.lftp > cdec.list
Result: https://gist.github.com/mojodna/c807e3fdbe618e99681aaef8e326f5e6
from joerd.
Some images are only 1 or 4px high, which triggered some problems in footprint generation (root cause: rasterio/rasterio#1025)
Transcoded versions in s3://mapzen-dynamic-tiler-test/cdem/0/
from joerd.
(I'm basically repeating what @mojodna said above but more explicitly. These were all run from an instance in AWS.)
Listing files
lftp \
-c 'find' \
ftp://ftp.geogratis.gc.ca/pub/nrcan_rncan/elevation/cdem_mnec/ \
| grep tif.zip > nrcan_cdem_listing.txt
Mirroring files to openterrain S3
cat nrcan_cdem_listing.txt \
| xargs \
-I {} \
-P 24 \
-n 1 \
sh -c 'export f="{}"; export s3="s3://elevation-sources-prod/nrcan_cdem/`basename $f`"; curl -s $f | AWS_DEFAULT_PROFILE=openterrain aws s3 cp - $s3; echo $s3' \
tee nrcan_cdem_s3_objects.txt
Transcoding files
I got the nrcan_cdem_s3_objects.txt
file onto my local computer, where I have the setup to submit transcoding jobs and ran:
cat nrcan_cdem_s3_objects.txt | \
while read filename; do
bn=$(basename $filename)
make submit-job job=aws/sample-transcode-job.json.hbs input=${filename} output=s3://elevation-sources-transcoded/nrcan_cdem/$(sed 's/_tif\.zip//' <<< $bn) name=$(sed 's/\./_/' <<< $bn)
done
from joerd.
It seems like we downloaded an older version of this data. There's a newer, higher-resolution dataset at http://open.canada.ca/data/en/dataset/957782bf-847c-4644-a757-e383c0057995?
Download link for this dataset is http://ftp.maps.canada.ca/pub/elevation/dem_mne/highresolution_hauteresolution/dsm_mns/2m/
from joerd.
Let's track that newer data in a different issue: #152
from joerd.
Related Issues (20)
- Add Arctic/REMA DEM mosaics over the Polar/Icy regions
- Add NASA HiMAT DEMs
- Slovenia LIDAR dataset HOT 1
- Vertical Coordinate System HOT 1
- Hosting of 260px tiles (with overlapping) on aws? HOT 1
- GMTED being used erroneously at high zoom levels
- Wrong tiles in Istanbul
- Data resolution HOT 2
- DEM FRANCE 1 meter open licence HOT 3
- SwissBATHY3D
- Add swissALTI3D dataset
- What vertical reference is used in the tile data? HOT 5
- How to generate normal tiles from a local dem file?
- Not able to generate normal tiles
- Uncomplete data in Germany on zoom level 14 and 15
- Contours ?
- DEM MAPS
- 1m LiDAR DEM for British Columbia
- opening .hgt.gz files in QGIS HOT 1
- CORS expose x-amz-meta-x-imagery-sources header HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from joerd.