cogeotiff / rio-tiler Goto Github PK
View Code? Open in Web Editor NEWUser friendly Rasterio plugin to read raster datasets.
Home Page: https://cogeotiff.github.io/rio-tiler/
License: BSD 3-Clause "New" or "Revised" License
User friendly Rasterio plugin to read raster datasets.
Home Page: https://cogeotiff.github.io/rio-tiler/
License: BSD 3-Clause "New" or "Revised" License
ref: https://github.com/vincentsarago/terrain-tiler
mapbox-gl-js (mapbox/mapbox-gl-js#5286) or mapzen-tangram support rgb encoded elevation values.
here is how to create a mapbox-gl compatible rgb values:
https://github.com/vincentsarago/terrain-tiler/blob/master/app/elevation.py#L16-L21
and for mapzen:
https://github.com/tilezen/joerd/blob/85ed04e7c2267880fe4272015ac37f396d91e864/joerd/output/terrarium.py#L31-L91
from numpy.histogram docs:
bins : int or sequence of scalars or str, optional
If bins is an int, it defines the number of equal-width bins in the given range (10, by default). If bins is a sequence, it defines a monotonically increasing array of bin edges, including the rightmost edge, allowing for non-uniform bin widths.
New in version 1.11.0.
If bins is a string, it defines the method used to calculate the optimal bin width, as defined by histogram_bin_edges.
https://github.com/DHI-GRAS/terracotta is a new lib inspired by rio-tiler made by @dionhaefner et all.
This lib has plenty of nice tricks which could be useful in rio-tiler
eg:
Hi @vincentsarago 👋
It happened, AWS now requires a requester pays header for Sentinel2 L1C 😢 . How do we move forward with rio tiler
from here?
This is the message I get when running the first example in the README
from rio_tiler import sentinel2
tile, mask = sentinel2.tile('S2A_tile_20170729_19UDP_0', 77, 89, 8)
# rasterio._err.CPLE_AWSAccessDeniedError: Access Denied
Interestingly when I run the command again I get this
# rasterio._err.CPLE_OpenFailedError: '/vsis3/sentinel-s2-l1c/tiles/19/U/DP/2017/7/29/0/preview.jp2' does not exist in the file system, and is not recognized as a supported dataset name.
which is more inline with what my (previously working) lambda sentinel2 tiler is now saying..
There is a bug in rio_tiler.utils.expression
that is not allowing band 8A
https://github.com/mapbox/rio-tiler/blob/d8a3581158a06511d23d58c9401244ea8ec6ee33/rio_tiler/utils.py#L552
https://github.com/cogeotiff/rio-tiler/search?q=max_workers&unscoped_q=max_workers
We have different values set in the code, but I think this is unnecessary and should only set one value and it should be customizable!
rio-tiler
should not be design to open a raster file every time it need to returns a tile (lambda function case) but we should be able to pass an open dataset and get the tile data.
rio_tiler.main.tile
: https://github.com/mapbox/rio-tiler/blob/91a63d5df0731e17a134e6595abb1ced4c7a3e29/rio_tiler/main.py#L39rio_tiler.utils.tile_band_worker
https://github.com/mapbox/rio-tiler/blob/91a63d5df0731e17a134e6595abb1ced4c7a3e29/rio_tiler/utils.py#L129Hey @vincentsarago
Hoping you can help me out by sharing a bit of knowledge (you're very good at it!)
I'm trying to tidy up a few loose ends on my geotiff-server and one of the things that's given me a bit of grief is colour correction, building my own version of this repo has been a good crash course in imagery & colours!
I think I understand your approach but was hoping you could validate that I vaguely understand
Before making any calls to get tiles the web app sends a request to the server to get some generic information that'll be used for subsequent tiling
metadata
endpointDuring this metadata call the server retrieves a 1024px x 1024px overview of each band for the entire geographic range of the scene
You then get the percentiles for each band and sends them back to the web app (defaults to 2% and 98% to remove the extremes)
nodata
pixelsAm I vaguely on track with my understanding?
Thanks,
Rowan
as mentioned in #54 (comment), rio-tiler should raise an error when the band names are not valid
Hello,
Newbie here. I've just installed rio-tiler
successfully, and I'm just trying out the examples on the README.md
:
(sat) [~] $ ipython
Python 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 18:15:35)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.2.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from rio_tiler import sentinel2
In [2]: tile, mask = sentinel2.tile('S2A_tile_20170729_19UDP_0', 77, 89, 8)
---------------------------------------------------------------------------
CPLE_AppDefinedError Traceback (most recent call last)
rasterio/_base.pyx in rasterio._base.DatasetBase.__init__()
rasterio/_shim.pyx in rasterio._shim.open_dataset()
rasterio/_err.pyx in rasterio._err.exc_wrap_pointer()
CPLE_AppDefinedError: Failed to connect to 169.254.169.254 port 80: Connection refused
During handling of the above exception, another exception occurred:
RasterioIOError Traceback (most recent call last)
<ipython-input-2-23a3f4efe0e1> in <module>
----> 1 tile, mask = sentinel2.tile('S2A_tile_20170729_19UDP_0', 77, 89, 8)
~/miniconda3/envs/sat/lib/python3.7/site-packages/cachetools/func.py in wrapper(*args, **kwargs)
74 except KeyError:
75 stats[1] += 1
---> 76 v = func(*args, **kwargs)
77 try:
78 with lock:
~/miniconda3/envs/sat/lib/python3.7/site-packages/rio_tiler/sentinel2.py in tile(sceneid, tile_x, tile_y, tile_z, rgb, r_bds, g_bds, b_bds, tilesize)
126
127 sentinel_preview = '{}/preview.jp2'.format(sentinel_address)
--> 128 with rasterio.open(sentinel_preview) as src:
129 wgs_bounds = transform_bounds(
130 *[src.crs, 'epsg:4326'] + list(src.bounds), densify_pts=21)
~/miniconda3/envs/sat/lib/python3.7/site-packages/rasterio/env.py in wrapper(*args, **kwds)
419
420 with env_ctor(session=session):
--> 421 return f(*args, **kwds)
422
423 return wrapper
~/miniconda3/envs/sat/lib/python3.7/site-packages/rasterio/__init__.py in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
214 # None.
215 if mode == 'r':
--> 216 s = DatasetReader(path, driver=driver, **kwargs)
217 elif mode == 'r+':
218 s = get_writer_for_path(path)(path, mode, driver=driver, **kwargs)
rasterio/_base.pyx in rasterio._base.DatasetBase.__init__()
RasterioIOError: Failed to connect to 169.254.169.254 port 80: Connection refused
In [3]:
Obviously, it has problem connecting to something but sure why. As I read it from the usage, one doesn't need to provide any credentials of any sort, correct? Any help would be most appreciated, thanks!!!
can be useful
We could maybe deprecate
Line 107 in 8c442e5
image_statistic
?) which in addition to percentiles, could also return min
, max
, mean
...Right now we apply linear color correction in the Landsat and Sentinel tile
functions
https://github.com/mapbox/rio-tiler/blob/f33930f4da0493c6941e5c3e617364eb9caca86a/rio_tiler/landsat8.py#L181-L188
and
https://github.com/mapbox/rio-tiler/blob/f33930f4da0493c6941e5c3e617364eb9caca86a/rio_tiler/sentinel2.py#L150-L158
IMO, the rescaling should be either in array_to_img
or handle in the handler
function (e.g https://github.com/mapbox/landsat-tiler/blob/27d8e870567ffa37727ea6287c4c4fcf0d4b7462/app/landsat.py#L80)
Doing this could able to create more high level functions.
I'm seing some resampling artifacts on tile
bottom border but only when the tile cover area with nodata.
The artifacts disappear when using nearest
resampling.
Lines 234 to 256 in 9618f13
also need to re-check #78
cc @sgillies (FYI)
Import works normal, package is installed correctly, but landsat8.metadata('LC08_L1TP_016037_20170813_20170814_01_RT', pmin=5, pmax=95)
gives me the following error.
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/cachetools/func.py", line 58, in wrapper
v = func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/rio_tiler/landsat8.py", line 71, in metadata
meta_data = utils.landsat_get_mtl(sceneid).get('L1_METADATA_FILE')
File "/usr/local/lib/python2.7/dist-packages/rio_tiler/utils.py", line 217, in landsat_get_mtl
raise Exception('Could not retrieve {} metadata'.format(sceneid))
Exception: Could not retrieve LC08_L1TP_016037_20170813_20170814_01_RT metadata
I am using Linux Ubuntu, Python 2.7.12.
there is no need to provide mapbox_elevation_rgb
and because rasterio has already a module to do so https://github.com/mapbox/rio-rgbify/blob/master/rio_rgbify/encoders.py#L4
"WarpedVRT does not permit boundless reads"
This is the error message I am getting with 1.0a7
If I change all boundless=True to boundless=False in utils.py, everything works again.
There is an example of how to convert it and apply it via pillow here. Happy to submit a PR if there is a generic approach that works.
Looks like one way to do it is make it an option on array_to_img
and then return the vrt source from tile_band_worker
and then aws.tile
. Then add a color_table_to_palette
helper function to utils. That way users that want to use the vrt color table can extract it and pass it to array_to_img
?
I guess another option is to use rasterio's write_colormap
but it looks like that would require writing the data out to vsimem and reading it back? Would that be better than applying it to the PNG directly?
(Also thanks for this repo :) I was struggling to get my own WarpedVRT code to work, then I found this)
in https://github.com/mapbox/rio-tiler/blob/e8caf27e7f3ad299c1523581173bb6d4d4be8977/rio_tiler/utils.py#L43 or https://github.com/mapbox/rio-tiler/blob/e8caf27e7f3ad299c1523581173bb6d4d4be8977/rio_tiler/utils.py#L99 we use input width=1024, height=1024
to tell rasterio to read the data overviews. While this work it can result in fetching more data than we really need.
With #90 @DanSchoppe added a third colormap
to rio-tiler. While this is awesome, I think rio-tiler could do more for allowing user to define custom colormap.
colormap are used when translating a one band data array to the image format using the new array_to_image
(using GDAL using a simple translation:
Line 386 in e512d84
So here ☝️ a rio_tiler.utils.array_to_image
compatible colormap object is usually an array of shape (256, 3)
c = get_colormap(format="gdal")
print(c.shape)
(256, 3)
print(c)
[
[255 255 255]
[250 250 250]
[246 246 246]
[242 242 242]
...
[255 1 191]
[255 1 207]
[255 1 223]
[255 1 239]
]
I first though we could have used a custom dict like
{
0: [255, 255, 255],
100: [255, 1, 191],
...
}
to create a colormap fordiscrete
values, but dict are not hashable so the current code in rio_tiler.utils.array_to_image
won't work.
I do love the dict option for colormap (like Rasterio use https://github.com/mapbox/rasterio/blob/960a906dad2a4e426387ce048a52c6e90afdcd2b/docs/topics/color.rst#writing-colormaps) so I first check if we can maybe change the rio_tiler.utils.get_colormap
function to have to return a dictionary colormap (because it seems to be the default format for GDAL) and in the same time adapt the rio_tiler.utils.array_to_image
to work with dict.
In a second time we could add another utility function to create colormaps, it seems that matplotlib has some functionality but I'm not sure we should had more required module to rio-tiler.
It looks like the rasterio release from today is breaking rio-tiler's functionality on my geotifs (which used to work before): https://github.com/mapbox/rasterio/releases/tag/1.0b2 Here is the error:
/tmp/rio/.env/lib/python3.5/site-packages/rio_tiler/utils.py:162: RasterioDeprecationWarning: dst_crs will
be removed after 1.0, use crs
with WarpedVRT(source, **vrt_params) as vrt:
Traceback (most recent call last):
File "/tmp/rio/tiler.py", line 49, in <module>
main()
File "/tmp/rio/tiler.py", line 41, in main
data, _ = tiler.tile(args.path, x, y, z)
File "/tmp/rio/.env/lib/python3.5/site-packages/rio_tiler/main.py", line 75, in tile
nodata=nodata, alpha=alpha)
File "/tmp/rio/.env/lib/python3.5/site-packages/rio_tiler/utils.py", line 168, in tile_read
indexes=indexes)
File "rasterio/_warp.pyx", line 979, in rasterio._warp.WarpedVRTReaderBase.read
ValueError: WarpedVRT does not permit boundless reads
Workaround
pip install 'rasterio==1.0b1' 'rio-tiler==1.0a7'
Small self-contained reproducible program
https://gist.github.com/daniel-j-h/69a967d63d833f74a123e2540bd146b9
Geotifs are from OpenAerialMap, e.g.
https://oin-hotosm.s3.amazonaws.com/5ac7745591b5310010e0d49a/0/5ac7745591b5310010e0d49b.tif
Let me know if I'm doing something wrong here or if this is an issue with rasterio.
Having those could be useful IMO
with rio-tiler==1.0a2
retrieving CBERS tiles is way slower and need more memory.
I wonder if reading a mask while we know it's not needed could be the reason.
https://github.com/mapbox/rio-tiler/blob/6100616a6b4a56e7778cf616da4c909530cc6126/rio_tiler/utils.py#L165-L168
Note: For Landsat/Sentinel we don't need to read the masks because there is no internal nodata and need to create the mask manually.
Features to develop for next versions:
GDAL/Rasterio is more efficient for image encoding than Pillow.
Note: Also removing Pillow will make lambda function lighter ;-)
The Sentinel-2 sceneid as defined in sat-api in in forme of S2A_tile_20171203_55MGS_0
where:
55
is the utm_zoneM
is the latitude_bandGS
is the grid_squarein https://github.com/mapbox/rio-tiler/blob/master/rio_tiler/utils.py#L293 the sceneid is defined as:
https://github.com/mapbox/rio-tiler/blob/f1c3d5f86650158707fad5843d736d3e9edb082f/rio_tiler/utils.py#L308-L310
in .main
we use rgb
option to pass band indexes. If we want to use the same terminology as in rasterio we should use indexes
.
For .sentinel2
, .landsat8
and cbers
I feel we should use the word bands
(!= indexes, because we read individual band files) instead of rgb
.
Hey Vincent,
I hit a CURL error (already fixed) but thought I should mention it here. This is for Linux Mint 18.1 Serena.
This worked
>>> from rio_tiler import landsat8
>>> bounds = landsat8.bounds('LC08_L1TP_016037_20170813_20170814_01_RT')
# {'sceneid': 'LC08_L1TP_016037_20170813_20170814_01_RT', 'bounds': [-81.30836, 32.10539, -78.82045, 34.22818]}
However getting metadata and tile both required curl (it seems), I hit this error:
rasterio._err.CPLE_HttpResponseError: CURL error: error setting certificate verify locations: CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none
I made sure that I had the ca-certificates
package
apt-get install ca-certificates
(the latest version was already installed). Then I fixed the CURL certificates path in .bashrc
export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
Everything now seems to work fine =D !
It was a bit weird (for me) that curl was looking in the wrong place (i.e. /etc/pki/tls/certs/ca-bundle.crt
), not sure how to proceed (e.g. note in docs, fix in code, ignore)?
I'm getting inconsistent and unhelpful rasterio errors whenever I try to use this library. Running the following code:
landsat8.tile("LC08_L1TP_221078_20180625_20180626_01_RT", 11825, 18900, 15)
first gives me the error:
rio_tiler/utils.py in tile_read(source, bounds, tilesize, indexes, nodata)
222
223 else:
--> 224 with rasterio.open(source) as src:
225 vrt_transform, vrt_width, vrt_height = get_vrt_transform(src,
226 bounds)CPLE_HttpResponseError: CURL error: error setting certificate verify locations: CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none
Running the same command again gives me
rio_tiler/utils.py in tile_read(source, bounds, tilesize, indexes, nodata)
222
223 else:
--> 224 with rasterio.open(source) as src:
225 vrt_transform, vrt_width, vrt_height = get_vrt_transform(src,
226 bounds)RasterioIOError: '/vsis3/landsat-pds/c1/L8/221/078/LC08_L1TP_221078_20180625_20180626_01_RT/LC08_L1TP_221078_20180625_20180626_01_RT_B4.TIF' does not exist in the file system, and is not recognized as a supported dataset name.
I get this behavior on both Sentilel and Landsat. I get this on all versions of rio_tiler, ranging from 0.0.3 to 1.0rc1
I've already tried the solutions in #52 and rasterio/rasterio#1289; neither helps.
The mask seems to be off and there might be something going on here https://github.com/mapbox/rio-tiler/blob/8ab4273dd5cd09af82e34f1e7bc37a89ef29d167/rio_tiler/utils.py#L197-L201
rio-tiler landsat8 should be able to return TIR Brightness Temperature
There is something strange going on with the pan
option in rio-tiler.landsat8.tile
https://github.com/cogeotiff/rio-tiler/blob/master/rio_tiler/landsat8.py#L397-L417
with pan-sharpening
without pan-sharpening
https://gist.github.com/vincentsarago/0ba79fbcfd9de7218a793e1e2c7c5823
HI Vincent,
Thanks again for your awesome medium article. It has inspired me to check out AWS. The first link in that article sent me here. Following the example to get a Landsat tile
>>> from rio_tiler import landsat8
>>> tile = landsat8.tile('LC08_L1TP_016037_20170813_20170814_01_RT', 71, 102, 8)
Looks like I need to set my AWS access key.
AttributeError: 'NoneType' object has no attribute 'access_key'
I'm a complete newb at AWS so don't know how to do that. Could you provide instructions in the README.md?
(if I find out I'll make a PR)
Thanks! ^_^
This project needs better documentation
I'm seeing a huge performance downgrade for the WarpedVRT
read after updating rasterio to version 1.0.18.
I've narrowed down, and the performance downgrade seems to be happening between version 1.0.13 and 1.0.14
I've tested with this dataset https://s3-us-west-2.amazonaws.com/remotepixel-pub/cog/LC08_L1TP_232064_20170806_20170813_01_T1_RGB654_cogeo.tif
Note: when using a dataset in `EPSG:4326: I'm not seeing the performance downgraded.
cc @sgillies
When we first started to work on rio-tiler it was mainly to access Landsat-8, but since we added Sentinel, CBERS and main
for any other COGs.
In https://github.com/cogeotiff/rio-tiler/blob/master/rio_tiler/utils.py we have plenty of utility functions which are specific to AWS PDS dataset and I feel each of this should be with their own parent
e.g
rio_tiler.utils.landsat_min_max_worker
-> rio_tiler.landsat8.landsat_min_max_worker
rio_tiler.utils.sentinel_parse_scene_id
-> rio_tiler.sentinel2.parse_scene_id
This will be a breaking change so if no one raise a hand, I'll move forward with this and add it to 1.0.0
Milestone
Each tile
function takes scene, x, y, z
as required inputs but rio_tiler.utils.expression
need expr
as well.
I think changing the type to keyword might ease the usability and enable to seamlessly create application that can use either expression
or {module}.tile
e.g
These new datasets are being experimentally ingested into CBERS on AWS repository.
Right now every bounds/metadata/tile functions have a @lru_cache()
decorator from cachetools.
This was made to have control over caching in the lambda function, but rio-tiler
could be use somewhere else than in a lambda function so we should remote it. Not really sure if there will be a performance improvement ro decrease, I'll try to do some benchmark.
We just published rio-tiler 1.0a0.
This version includes some new features like:
Here are some next point I'd like to focus before releasing a final v1:
create mask inside the tile
functions (instead of defining it in array_to_img). Basically the tile function would return a tile data
and a mask
which could be used in array_to_img
. This will able to rescale data between 0 and 255 (instead of 1 -> 255, 0 being used as nodata).
Automatic color correction instead of linear rescaling (using rio-color).
better support for main
(local or https) and s3
files.
Reopening issue #53 because this has not been fixed and is an issue with rio-tiler.
Running the following:
from rio_tiler import sentinel2
import os
os.environ["CURL_CA_BUNDLE"] = "/etc/ssl/certs/ca-certificates.crt"
os.environ["AWS_REQUEST_PAYER"] = "requester"
tile, mask = sentinel2.tile("S2B_tile_20180813_23MQQ_0", 781, 1052, 11, (4, 3, 2), tilesize=256)
gives the error:
RasterioIOError: '/vsis3/sentinel-s2-l1c/tiles/23/M/QQ/2018/8/13/0/B4.jp2' does not exist in the file system, and is not recognized as a supported dataset name.
running gdalinfo on this path gives the error:
ERROR 4: /vsis3/sentinel-s2-l1c/tiles/23/M/QQ/2018/8/13/0/B4.jp2: No such file or directory
gdalinfo failed - unable to open '/vsis3/sentinel-s2-l1c/tiles/23/M/QQ/2018/8/13/0/B4.jp2'.
This path is being generated by rio-tiler. This is only happening on Sentinel; Landsat works fine.
I use python 3.6 and installed rio-tiler from repo.
And script run time is too high.
from aws_sat_api.search import landsat
from rio_tiler import landsat8
from datetime import datetime
startTime = datetime.now()
l8_path = 178
l8_row = 21
full_search = False
l8_meta = landsat(l8_path, l8_row, full_search)
metadata = landsat8.metadata('LC81780212013117LGN01', 5, 95)
print(metadata)
print(datetime.now() - startTime)
Output:
{'sceneid': 'LC81780212013117LGN01', 'bounds': [36.11826, 54.83456, 40.03735, 56.95001], 'rgbMinMax': {'1': [1489, 7031], '2': [1238, 7094], '3': [922, 6769], '4': [777, 7117], '5': [1278, 7794], '6': [811, 6038], '7': [474, 4849], '9': [17, 336], '10': [264, 285], '11': [261, 281]}}
0:03:00.593763
Please help.
There is a bug in rasterio (mentioned in rasterio/rasterio#1191, rasterio/rasterio#1266) WrappedVrt which might be linked in poor performance when reading mask
withing a file.
For sentinel, landsat and CBERS we don't read the mask because we are creating one using a nodata value.
Calling array_to_image
using the array and mask returned from tile
using a 4 band tiff returns an error: WEBP driver doesn't support 5 bands. Must be 3 (RGB) or 4 (RGBA) bands.
# image at URL is 4 band geotiff
tile, mask = main.tile(url, int(tile_x), int(tile_y), int(tile_z), tilesize=256, nodata=None)
options = img_profiles["webp"]
buffer = utils.array_to_image(tile, mask=mask, img_format="webp", color_map=None, **options)
It looks like utils:392 is just adding a band to the array without checking to see whether an alpha band already exists
let's add pre-commit
and black
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.