Comments (16)
Getting the download URLs looks like this:
curl -s http://www.geostore.com/environment-agency/rest/product/EA_SUPPLIED_OS_10KM/SP16?catalogName=Survey \
| jq -r '.[] | select(.coverageLayer | contains("LIDAR-DSM-1M-ENGLAND-EA-MD-YY")) | "http://www.geostore.com/environment-agency/rest/product/download/\(.guid)\t\(.fileName)"'
This returns a TSV with the URL and the filename:
http://www.geostore.com/environment-agency/rest/product/download/c7e2fa0c-157d-11e7-a00a-8cdcd4b4861c LIDAR-DSM-1M-SP16ne.zip
http://www.geostore.com/environment-agency/rest/product/download/408e5664-157d-11e7-a00a-8cdcd4b4861c LIDAR-DSM-1M-SP16sw.zip
http://www.geostore.com/environment-agency/rest/product/download/4829effa-157d-11e7-a00a-8cdcd4b4861c LIDAR-DSM-1M-SP16nw.zip
http://www.geostore.com/environment-agency/rest/product/download/c3af14b6-157d-11e7-a00a-8cdcd4b4861c LIDAR-DSM-1M-SP16se.zip
from joerd.
From @kevinkreiser:
@robhawkes yes as many suggestions as you can muster would be great. we don't currently have a schedule for adding new datasets, but once we have a decent list going we can start to do that. as you point out, with each new data set we'll have to jump through the hoops to get it into the proper format but that's part of the fun ;)
one question i have for you though. what kind of horizontal resolution are you looking for? Currently we are doing 1 arc second a la srtm (lower res datasets are upscaled). in terms of disk space, much higher than that becomes prohibitive (when you are dealing with the whole world). we could start to do variable resolution tiles internally but the complications that brings (and possibly decrease in performance) is probably not worth it. so if we integrate this, would you be expecting 25cm resolution or could you make due with something more like 10 meters?
from joerd.
This would be better data than the 30m EU data in #132.
from joerd.
I put together a script to scrape together all the URLs:
https://github.com/iandees/uk-lidar
The catalog.csv
file has tens of thousands of lines in it, but we only need some of them. Either way, this source is broken into a whooole lot of files so it's going to be quite a bit of work to transcode.
from joerd.
Starting on 2m terrain model, looks like there are 5,679 images to grab:
curl -s https://raw.githubusercontent.com/iandees/uk-lidar/master/catalog.csv \
| grep LIDAR-DTM-2M-ENGLAND-EA-MD-YY \
| wc -l
Mirroring data
curl -s https://raw.githubusercontent.com/iandees/uk-lidar/master/catalog.csv \
| grep LIDAR-DTM-2M-ENGLAND-EA-MD-YY \
| cut -d, -f 8,5 \
| xargs -I {} -P 24 -n 1 \
sh -c 'export f="{}"; a=$(cut -d, -f 1 <<< $f); b=$(cut -d, -f 2 <<< $f); s3=s3://elevation-sources-prod/uk_lidar/$a; curl -s http://www.geostore.com/environment-agency/rest/product/download/$b | AWS_PROFILE=openterrain aws s3 cp - $s3 && echo $s3' \
| tee uk_lidar_s3_objects.txt
from joerd.
At this point I tried opening one of these files to verify that the transcode process will be able to handle it. It turns out each of the .zip files has ~a dozen or more .asc
files. A couple hiccups as a result:
- The transcode system doesn't know how to handle
.asc
or ASCII files. gdal and qgis know what to do, but you have to manually pair a CRS with the data, which the transcode system can't do easily right now. (We'd have to give each image a VRT with the projection information embedded in it) - The transcode system currently only knows how to handle one image at a time. When it encounters a .zip or .tar.gz, it looks for the first .tif or .dem contained there-in and processes that.
I'd appreciate @mojodna's thoughts on this, but I think it would make sense to come up with an external process that composites these tiny little slices of DEM back into a larger TIF that is more suitable for consumption by our existing transcode system. This recombobulated data would probably be useful to others trying to consume this data from the UK, too.
from joerd.
from joerd.
I think it would make sense to come up with an external process that composites these tiny little slices of DEM back into a larger TIF that is more suitable for consumption by our existing transcode system. This recombobulated data would probably be useful to others trying to consume this data from the UK, too.
Absolutely! The tiles alone will be incredibly useful but if there was some way of getting a combined set of elevation data for the UK that would be amazing.
from joerd.
Over the weekend I added a few scripts to my uk-lidar repo to assist with this. I processed the .zip's of .asc's into .tif's and started merging them into larger images, but it looks like I accidentally deleted the results of that work.
I'm re-running that process now and will get started transcoding when the merging step is done.
from joerd.
I completed the process to convert, merge, and upload the 2m LIDAR-based data from the UK and documented it here:
https://github.com/iandees/uk-lidar/blob/master/dem_composite/README.md
Transcode files
aws s3 ls s3://elevation-sources-prod/uk_lidar/ --recursive | \
grep .tif$ | \
awk '{print $4}' | \
while read filename; do \
bn=$(basename ${filename%.*}); \
make submit-job job=aws/transcode-job.json.hbs input=s3://elevation-sources-prod/${filename} output=s3://elevation-sources-transcoded/$(dirname $filename)/${bn} name=${bn:0:50}
done
from joerd.
I finished transcoding and loading footprints to get this:
The data looked great, but there was a large chunk of data missing in the lower right. I re-downloaded and re-processed the dataset to pull in that missing data and got this after re-doing the transcode step:
One important thing to point out is that this dataset is not 100% coverage of the UK like the images above suggest. It appears that they collected this data on seemingly random flights in different directions over the course of time:
From the dataset preview map
This means there will be bits of SRTM poking through in places where they haven't collected data yet.
from joerd.
This looks amazing! The LIDAR data definitely isn't 100% coverage (more like 70%) and the coverage also depends on the year that the data was collected, at least from what I can tell. I imagine many of the flights are near some kind of river or body of water (for flood risk analysis), giving the seemingly random direction of flights.
Here's the Environment Agency description of the data:
The Environment Agency’s LIDAR data archive contains digital elevation data derived from surveys carried out by the Environment Agency's specialist remote sensing team. Accurate elevation data is available for over 70% of England. This dataset is derived from a combination of our full dataset which has been merged and re-sampled to give the best possible coverage. Data is available at 2m, 1m, 50cm, and 25cm resolution.
There's also LIDAR data for Wales and Scotland's is due any time now. Shall I create new issues for them?
from joerd.
There's also LIDAR data for Wales and Scotland's is due any time now. Shall I create new issues for them?
Yes! The more data the better.
from joerd.
from joerd.
It appears that Defra have moved their data onto a new platform at https://environment.data.gov.uk/. Has anyone tried to bulk download from there? I tried but failed miserably!
from joerd.
I've been poking at it for a while and haven't found a nice way to do it.
from joerd.
Related Issues (20)
- Add Arctic/REMA DEM mosaics over the Polar/Icy regions
- Add NASA HiMAT DEMs
- Slovenia LIDAR dataset HOT 1
- Vertical Coordinate System HOT 1
- Hosting of 260px tiles (with overlapping) on aws? HOT 1
- GMTED being used erroneously at high zoom levels
- Wrong tiles in Istanbul
- Data resolution HOT 2
- DEM FRANCE 1 meter open licence HOT 3
- SwissBATHY3D
- Add swissALTI3D dataset
- What vertical reference is used in the tile data? HOT 5
- How to generate normal tiles from a local dem file?
- Not able to generate normal tiles
- Uncomplete data in Germany on zoom level 14 and 15
- Contours ?
- DEM MAPS
- 1m LiDAR DEM for British Columbia
- opening .hgt.gz files in QGIS HOT 1
- CORS expose x-amz-meta-x-imagery-sources header HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from joerd.