Comments (6)
I did a highly unscientific single test case, but I think it's fairly representative of the whole; take NED13 n32w090
, which is native Float32
data type:
- 359MB - Original
.img
compressed in a.zip
file from USGS. - 447MB - Uncompressed
.img
- 266MB - GeoTiff as
Float32
withCOMPRESS=DEFLATE, PREDICTOR=3
- 28MB - GeoTIFF as
Int16
withCOMPRESS=LZW, PREDICTOR=2
The reduction in file size by almost an order of magnitude is a benefit that has to be weighed against the improvement in precision allowed by Float32
. Just out of interest I tried:
- Using feet instead of meters, if we can accept some clipping in the deepest parts of the ocean: 47MB
- Using cm instead of meters, and
Int32
data type to avoid clipping: 182MB
Of course, the files will be larger, as we're retaining more of the information from the original. Sub-meter information can be very useful, for example in flat areas to avoid quantization artefacts appearing as false ridges, and many other uses. But its presence in the file comes at a cost for users who don't need that level of precision. It's not clear to me where the balance point lies.
Side note
Although it should be noted that not all of the data in the sub-meter range is information. For example, in tile 13/1326/3123
which, in the "terrarium" format, retains sub-meter information down to 1/256th of a meter in the blue band:
The "starburst" pattern is clearly an artefact, perhaps from merging multiple DEMs around a reference point. So although there's precision in NED, the accuracy should be carefully evaluated. LIDAR, likewise, has the capability of being incredibly precise, but care should be taken to avoid assumptions about its accuracy.
from joerd.
@nvkelso, what do you think about the precision / file size trade-off? We could, of course, make a separate format for Int16
and Float32
GeoTiffs?
from joerd.
What about generating the Int16
files for most zooms, but switching to Float32
at zoom 14/15+?
from joerd.
I suspect that that will confuse the GDAL WMS driver, since it treats the whole thing as a single source with overviews.
from joerd.
No, I tried, GDAL is fine with specifying a pixel type that is different from the data pixel type.
This works:
<GDAL_WMS> <Service name="TMS"> <ServerUrl>https://s3.amazonaws.com/elevation-tiles-prod/geotiff/${z}/${x}/${y}.tif</ServerUrl> </Service> <DataWindow> <UpperLeftX>-20037508.34</UpperLeftX> <UpperLeftY>20037508.34</UpperLeftY> <LowerRightX>20037508.34</LowerRightX> <LowerRightY>-20037508.34</LowerRightY> <TileLevel>14</TileLevel> <TileCountX>1</TileCountX> <TileCountY>1</TileCountY> <YOrigin>top</YOrigin> </DataWindow> <Projection>PROJCS["WGS_1984_Web_Mercator", GEOGCS["GCS_WGS_1984_Major_Auxiliary_Sphere", DATUM["WGS_1984_Major_Auxiliary_Sphere", SPHEROID["WGS_1984_Major_Auxiliary_Sphere",6378137.0,0.0]], PRIMEM["Greenwich",0.0], UNIT["Degree",0.0174532925199433]], PROJECTION["Mercator_1SP"], PARAMETER["False_Easting",0.0], PARAMETER["False_Northing",0.0], PARAMETER["Central_Meridian",0.0], PARAMETER["latitude_of_origin",0.0], UNIT["Meter",1.0], EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs"], AUTHORITY["EPSG","102113"]]</Projection> <BlockSizeX>512</BlockSizeX> <BlockSizeY>512</BlockSizeY> <BandsCount>1</BandsCount> <DataType>Float32</DataType> <UnsafeSSL>true</UnsafeSSL> </GDAL_WMS>
Regarding compression: Rounding the elevation values to one decimal for zoom 14, maybe 2 for zoom 15+ will decrease size quite a bit. I think this would be the better solution than switching to cm and int32.
from joerd.
@flitzi different pixel types at varying zooms? That's fantastic.
from joerd.
Related Issues (20)
- GMTED being used erroneously at high zoom levels
- Wrong tiles in Istanbul
- Data resolution HOT 2
- DEM FRANCE 1 meter open licence HOT 3
- SwissBATHY3D
- Add swissALTI3D dataset
- What vertical reference is used in the tile data? HOT 5
- How to generate normal tiles from a local dem file?
- Not able to generate normal tiles
- Uncomplete data in Germany on zoom level 14 and 15
- Contours ?
- DEM MAPS
- 1m LiDAR DEM for British Columbia
- opening .hgt.gz files in QGIS HOT 1
- CORS expose x-amz-meta-x-imagery-sources header HOT 3
- contours.axismaps.com HOT 1
- EU AWS S3 bucket HOT 2
- S3 bucket download to linux issue HOT 7
- Bad data in 14,2519,5484 HOT 2
- Bad data around 15/19444/12981 and westwards
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from joerd.