Open lidar-based DSM and DTM dataset for ~72% of UK about joerd HOT 16 OPEN

Getting the download URLs looks like this:

curl -s http://www.geostore.com/environment-agency/rest/product/EA_SUPPLIED_OS_10KM/SP16?catalogName=Survey \
  | jq -r '.[] | select(.coverageLayer | contains("LIDAR-DSM-1M-ENGLAND-EA-MD-YY")) | "http://www.geostore.com/environment-agency/rest/product/download/\(.guid)\t\(.fileName)"'

This returns a TSV with the URL and the filename:

http://www.geostore.com/environment-agency/rest/product/download/c7e2fa0c-157d-11e7-a00a-8cdcd4b4861c   LIDAR-DSM-1M-SP16ne.zip
http://www.geostore.com/environment-agency/rest/product/download/408e5664-157d-11e7-a00a-8cdcd4b4861c   LIDAR-DSM-1M-SP16sw.zip
http://www.geostore.com/environment-agency/rest/product/download/4829effa-157d-11e7-a00a-8cdcd4b4861c   LIDAR-DSM-1M-SP16nw.zip
http://www.geostore.com/environment-agency/rest/product/download/c3af14b6-157d-11e7-a00a-8cdcd4b4861c   LIDAR-DSM-1M-SP16se.zip

from joerd.

From @kevinkreiser:

@robhawkes yes as many suggestions as you can muster would be great. we don't currently have a schedule for adding new datasets, but once we have a decent list going we can start to do that. as you point out, with each new data set we'll have to jump through the hoops to get it into the proper format but that's part of the fun ;)

one question i have for you though. what kind of horizontal resolution are you looking for? Currently we are doing 1 arc second a la srtm (lower res datasets are upscaled). in terms of disk space, much higher than that becomes prohibitive (when you are dealing with the whole world). we could start to do variable resolution tiles internally but the complications that brings (and possibly decrease in performance) is probably not worth it. so if we integrate this, would you be expecting 25cm resolution or could you make due with something more like 10 meters?

from joerd.

This would be better data than the 30m EU data in #132.

from joerd.

I put together a script to scrape together all the URLs:

https://github.com/iandees/uk-lidar

The catalog.csv file has tens of thousands of lines in it, but we only need some of them. Either way, this source is broken into a whooole lot of files so it's going to be quite a bit of work to transcode.

from joerd.

Starting on 2m terrain model, looks like there are 5,679 images to grab:

curl -s https://raw.githubusercontent.com/iandees/uk-lidar/master/catalog.csv \
 | grep LIDAR-DTM-2M-ENGLAND-EA-MD-YY \
 | wc -l

Mirroring data

curl -s https://raw.githubusercontent.com/iandees/uk-lidar/master/catalog.csv \
 | grep LIDAR-DTM-2M-ENGLAND-EA-MD-YY \
 | cut -d, -f 8,5 \
 | xargs -I {} -P 24 -n 1 \
  sh -c 'export f="{}"; a=$(cut -d, -f 1 <<< $f); b=$(cut -d, -f 2 <<< $f); s3=s3://elevation-sources-prod/uk_lidar/$a; curl -s http://www.geostore.com/environment-agency/rest/product/download/$b | AWS_PROFILE=openterrain aws s3 cp - $s3 && echo $s3' \
 | tee uk_lidar_s3_objects.txt

from joerd.

At this point I tried opening one of these files to verify that the transcode process will be able to handle it. It turns out each of the .zip files has ~a dozen or more .asc files. A couple hiccups as a result:

• The transcode system doesn't know how to handle .asc or ASCII files. gdal and qgis know what to do, but you have to manually pair a CRS with the data, which the transcode system can't do easily right now. (We'd have to give each image a VRT with the projection information embedded in it)
• The transcode system currently only knows how to handle one image at a time. When it encounters a .zip or .tar.gz, it looks for the first .tif or .dem contained there-in and processes that.

I'd appreciate @mojodna's thoughts on this, but I think it would make sense to come up with an external process that composites these tiny little slices of DEM back into a larger TIF that is more suitable for consumption by our existing transcode system. This recombobulated data would probably be useful to others trying to consume this data from the UK, too.

from joerd.

from joerd.

I think it would make sense to come up with an external process that composites these tiny little slices of DEM back into a larger TIF that is more suitable for consumption by our existing transcode system. This recombobulated data would probably be useful to others trying to consume this data from the UK, too.

Absolutely! The tiles alone will be incredibly useful but if there was some way of getting a combined set of elevation data for the UK that would be amazing.

from joerd.

Over the weekend I added a few scripts to my uk-lidar repo to assist with this. I processed the .zip's of .asc's into .tif's and started merging them into larger images, but it looks like I accidentally deleted the results of that work.

I'm re-running that process now and will get started transcoding when the merging step is done.

from joerd.

I completed the process to convert, merge, and upload the 2m LIDAR-based data from the UK and documented it here:
https://github.com/iandees/uk-lidar/blob/master/dem_composite/README.md

Transcode files

aws s3 ls s3://elevation-sources-prod/uk_lidar/ --recursive | \
grep .tif$ | \
awk '{print $4}' | \
while read filename; do \
    bn=$(basename ${filename%.*}); \
    make submit-job job=aws/transcode-job.json.hbs input=s3://elevation-sources-prod/${filename} output=s3://elevation-sources-transcoded/$(dirname $filename)/${bn} name=${bn:0:50}
done

from joerd.

I finished transcoding and loading footprints to get this:

The data looked great, but there was a large chunk of data missing in the lower right. I re-downloaded and re-processed the dataset to pull in that missing data and got this after re-doing the transcode step:

One important thing to point out is that this dataset is not 100% coverage of the UK like the images above suggest. It appears that they collected this data on seemingly random flights in different directions over the course of time:

From the dataset preview map

This means there will be bits of SRTM poking through in places where they haven't collected data yet.

from joerd.

This looks amazing! The LIDAR data definitely isn't 100% coverage (more like 70%) and the coverage also depends on the year that the data was collected, at least from what I can tell. I imagine many of the flights are near some kind of river or body of water (for flood risk analysis), giving the seemingly random direction of flights.

Here's the Environment Agency description of the data:

The Environment Agency’s LIDAR data archive contains digital elevation data derived from surveys carried out by the Environment Agency's specialist remote sensing team. Accurate elevation data is available for over 70% of England. This dataset is derived from a combination of our full dataset which has been merged and re-sampled to give the best possible coverage. Data is available at 2m, 1m, 50cm, and 25cm resolution.

There's also LIDAR data for Wales and Scotland's is due any time now. Shall I create new issues for them?

from joerd.

There's also LIDAR data for Wales and Scotland's is due any time now. Shall I create new issues for them?

Yes! The more data the better.

from joerd.

from joerd.

It appears that Defra have moved their data onto a new platform at https://environment.data.gov.uk/. Has anyone tried to bulk download from there? I tried but failed miserably!

from joerd.

I've been poking at it for a while and haven't found a nice way to do it.

from joerd.