Code Monkey home page Code Monkey logo

copernicus_tools's Introduction

Copernicus Global Land tools

This plugin allows to download all the Copernicus Land products in easy way and translate them to geotiff. This tool would like to be like a Swiss knife for the Copernicus Land Products.

To use this tool it is necessary to have a valid account to CGL website.

The plugin is built as "processing plugin", it is possible to find all the algorithms in the Processing Toolbox inside the "CGL_Provider". The two most used tools, the Copernicus Global Land Products Downloader and the Land Cover Downloader, are also available in the QGIS ToolBar.

To find more details and technical documentation about Copernicus Land Products, please visit the website

How to use the Copernicus Global Land Downloader

This algorithm allows download Copernicus Global Land products and converts the native Netcdf files into geotiff. Select the product collection to download and the day. The algorithm will download the product with the closest date. Download directory is the directory in which the product will be downloaded and converted to geotiff.

This tool allows users to connect to the manifest of CGL products and to choose which one the user wants to download. The algorithm downloads the NetCDF file and converts it to Geotiff.
To use this tool it is necessary to have a valid account to CGL (https://land.copernicus.eu/) website.

  • Select the product collection to download and the day. The algorithm will download the product with the closest date.
  • Download directory is the directory in which the product will be downloaded and converted to geotiff.

Downloader

In order to works this tool requires Pandas library. Usually, this library is installed within QGIS. If your installation doesn't come with Pandas, please install it trough OSGeo4W Installer or via pip.

How to use the Copernicus Global Land Downloader

The tool permit to download the Land Cover tiles from Copernicus Land Services.

CGL Resampler Tool

A PyQGIS script to download Copernicus Global Land Products and to resample 333m products to 1Km.

The CGLS vegetation-related products (i.e. NDVI, LAI, FAPAR…), based on PROBA-V observations, have been distributed at 1km and 333m spatial resolution until June, 2020. However, as of July, 2020, all Near Real Time (NRT) production of the vegetation biophysical variables, based on Sentinel-3 observations, are no longer provided at 1km resolution. Nonetheless, users interested in continuing their 1km time series can use a resample of the new 333m products. This algorithm allows to resample the 333meters Copernicus Products to 1Km preserving the spatial extension of 1Km time series.

First of all, in order to use this tool, is necessary to add in the Processing toolbox both the raster_calc_copernicus.py and CGL_resampler.py scripts.
This algorithm allows to resample the 333meters Copernicus Products to 1Km preserving the spatial extension of 1Km time series.
In order to perform this conversion, the following steps are needed:

  • Reclassify the input raster in 0-1 (where 1 are valid values),
  • Resample the output using mode -so 0 is where at least 5 pixels inside the kernel aren't valid -
  • Multiply these results with the result of 3x3 resampling using average.
    All these steps are needed in order to include the condition that at least 5 out of the 9 pixels have to have valid values (i.e. not NA) to return a valid value for the resampled pixel (333m to 1Km).

process

The UI of this algorithm is the following:

resampler_UI

  • Input Raster: the input raster to resample (NetCDF, Geotiff...)
  • Resampling method: It is possible to choose the resampling method to use. Possible values: nearest, bilinear,cubic,cubicspline,lanczos,average,mode (see GDAL_Translate options). Default value: average.
  • Reclassify valid data: this parameter is used to reclassify the input raster to exclude the not valid parameters. Default value: [-1,1,1,1,255,0] . The Default value is set for NDVI products. [min_valid_value <= max_valid_value --> 1, min_not_valid_data <= max_not_valid_data --> 0]
  • Output: the output file

See the following table to see which method is most suitable for each product:

resampling methods

The original Global Land product files can often contain specific values for invalid pixels (flagged values), which need to be dealt with.
In the case of the NDVI products, for example, digital values in the netCDF (DN) larger than 250 are flagged and need to be converted to NA (No Data).
When the netCDF files are read in as a raster object, the digital values are scaled into real NDVI values automatically (-0.08:0.93).
Therefore, after reading the files, all pixels with NDVI values larger than 0.92 (= 250 x scale + offset; in this case, scale = 0.004 and offset = -0.08) were set to NA.
In the same way, all the other products’ non-valid values were transformed to NAs according to their valid ranges, which can be seen in nex table.
In addition, other supporting information of each product can be found both in the netCDF file metadata and in their Product User Manual at https://land.copernicus.eu/global/products/.

See the Cut-off of valid values for each product/layer

Results of the resampling tool

The tool has been tested and the results have been compared with the Original 1Km series and with the results of the R Notebook
Read more about the Resampling Tool

Product Date Reclassify table Resampling method
NDVI 01/05/2019 [-1,1,1,1,255,0] Average
FAPAR 10/05/2019 [0,7,1,7,210,0] Average
LAI 10/05/2019 [-1,1,1,1,255,0] Average
FCOVER 10/05/2019 [0,1,1,1,250,0] Average
DMP 10/05/2019 [0,327.67,1,327.67,3267,0] Average
NDVI-Europe
Pearson's r Root Mean Square Error Mean Absolute Error
1 Original 1Km Product - R Notebook 0.97951 0.05048 0.03123
2 Original 1Km Product - QGIS Tool 0.97532 0.05476 0.03326
3 R Notebook - QGIS Tool 0.99999 0.00114 0.00098

QGIS_Original QGIS_R

LAI-Europe
Pearson's r Root Mean Square Error Mean Absolute Error
1 Original 1Km Product - R Notebook 0.9416 0.46936 0.31021
2 Original 1Km Product - QGIS Tool 0.94005 0.47589 0.31375
3 R Notebook - QGIS Tool 0.99995 0.01297 0.00737

QGIS_Original QGIS_R

LAI-Amazonia
Pearson's r Root Mean Square Error Mean Absolute Error
1 Original 1Km Product - R Notebook 0.70523 0.90322 0.65949
2 Original 1Km Product - QGIS Tool 0.70638 0.96189 0.68983
3 R Notebook - QGIS Tool 0.99988 0.01338 0.00834

QGIS_Original QGIS_R

FCOVER-Europe
Pearson's r Root Mean Square Error Mean Absolute Error
1 Original 1Km Product - R Notebook 0.97581 0.06967 0.04841
2 Original 1Km Product - QGIS Tool 0.97396 0.07195 0.04931
3 R Notebook - QGIS Tool 0.99998 0.00204 0.00095

QGIS_Original QGIS_R

FCOVER-Amazonia
Pearson's r Root Mean Square Error Mean Absolute Error
1 Original 1Km Product - R Notebook 0.70206 0.11937 0.0835
2 Original 1Km Product - QGIS Tool 0.71047 0.12503 0.0865
3 R Notebook - QGIS Tool 0.99988 0.00197 9e-04

QGIS_Original QGIS_R

FAPAR-W Africa
Pearson's r Root Mean Square Error Mean Absolute Error
1 Original 1Km Product - R Notebook 0.986 0.05055 0.02808
2 Original 1Km Product - QGIS Tool 0.98201 0.05913 0.03304
3 R Notebook - QGIS Tool 0.99999 0.00101 0.00063

QGIS_Original QGIS_R

FAPAR-Europe
Pearson's r Root Mean Square Error Mean Absolute Error
1 Original 1Km Product - R Notebook 0.97449 0.06302 0.04307
2 Original 1Km Product - QGIS Tool 0.9723 0.0655 0.04406
3 R Notebook - QGIS Tool 0.99998 0.0017 0.00096

QGIS_Original QGIS_R

FAPAR-Amazonia
Pearson's r Root Mean Square Error Mean Absolute Error
1 Original 1Km Product - R Notebook 0.67006 0.09832 0.0527
2 Original 1Km Product - QGIS Tool 0.68385 0.10462 0.0562
3 R Notebook - QGIS Tool 0.99993 0.00111 0.00087

QGIS_Original QGIS_R

DMP-W Africa
Pearson's r Root Mean Square Error Mean Absolute Error
1 Original 1Km Product - R Notebook 0.98686 5.32346 2.69047
2 Original 1Km Product - QGIS Tool 0.98407 6.04546 3.1398
3 R Notebook - QGIS Tool 1 0.02141 0.00264
DMP-Europe
Pearson's r Root Mean Square Error Mean Absolute Error
1 Original 1Km Product - R Notebook 0.9763 6.14813 4.11369
2 Original 1Km Product - QGIS Tool 0.97586 6.19854 4.13796
3 R Notebook - QGIS Tool 0.99999 0.0902 0.00527

QGIS_Original QGIS_R

copernicus_tools's People

Contributors

fgianoli avatar enricofer avatar

Stargazers

Aichi 2020 avatar Jacky Volpes avatar Tenz avatar

Watchers

 avatar

Forkers

enricofer

copernicus_tools's Issues

classFactory() error when installing cgl_tools-2.0.zip

I tried to install plugin cgl_tools-2.0.zip to QGIS from the zip file: error below.

Couldn't load plugin 'cgl_tools' due to an error when calling its classFactory() method

ImportError: cannot import name 'PROTOCOL_TLS' from 'urllib3.util.ssl_' (/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/urllib3/util/ssl_.py)
Traceback (most recent call last):
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 423, in _startPlugin
plugins[packageName] = package.classFactory(iface)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/init.py", line 49, in classFactory
from .CGL_tools import CGL_toolsPlugin
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/CGL_tools.py", line 58, in
from .CGL_tools_provider import CGL_toolsProvider
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/CGL_tools_provider.py", line 34, in
from .CGL_tools_algorithm import LandCoverDownload
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/CGL_tools_algorithm.py", line 60, in
import boto3
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/extlibs/boto3/init.py", line 17, in
from boto3.session import Session
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/extlibs/boto3/session.py", line 17, in
import botocore.session
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/extlibs/botocore/session.py", line 26, in
import botocore.client
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/extlibs/botocore/client.py", line 15, in
from botocore import waiter, xform_name
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/extlibs/botocore/waiter.py", line 18, in
from botocore.docs.docstring import WaiterDocstring
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/extlibs/botocore/docs/init.py", line 15, in
from botocore.docs.service import ServiceDocumenter
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/extlibs/botocore/docs/service.py", line 14, in
from botocore.docs.client import ClientDocumenter, ClientExceptionsDocumenter
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/extlibs/botocore/docs/client.py", line 14, in
from botocore.docs.example import ResponseExampleDocumenter
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/extlibs/botocore/docs/example.py", line 13, in
from botocore.docs.shape import ShapeDocumenter
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/extlibs/botocore/docs/shape.py", line 19, in
from botocore.utils import is_json_value_header
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in _import
mod = builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/extlibs/botocore/utils.py", line 34, in
import botocore.httpsession
File "/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/qgis/utils.py", line 888, in import
mod = builtin_import(name, globals, locals, fromlist, level)
File "/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/extlibs/botocore/httpsession.py", line 21, in
from urllib3.util.ssl
import (
ImportError: cannot import name 'PROTOCOL_TLS' from 'urllib3.util.ssl
' (/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/urllib3/util/ssl
.py)

Python version: 3.9.5 (default, Sep 10 2021, 16:18:19) [Clang 12.0.5 (clang-1205.0.22.11)]
QGIS version: 3.28.7-Firenze Firenze, 62cbdb06b6

Python Path:
/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/sense_detection
/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python
/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python
/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins
/Applications/QGIS-LTR.app/Contents/MacOS/../Resources/python/plugins
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/Fiona-1.8.13.post1-py3.9-macosx-10.13.0-x86_64.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/scipy-1.5.1-py3.9-macosx-10.13.0-x86_64.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/opencv_contrib_python-4.3.0.36-py3.9-macosx-10.13.0-x86_64.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/cftime-1.2.1-py3.9-macosx-10.13.0-x86_64.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python39.zip
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/Rtree-0.9.7-py3.9-macosx-10.13.0-x86_64.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/statsmodels-0.11.1-py3.9-macosx-10.13.0-x86_64.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/pyproj-3.2.0-py3.9-macosx-10.13.0-x86_64.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/lib-dynload
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/rasterio-1.1.5-py3.9-macosx-10.13.0-x86_64.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/numpy-1.20.1-py3.9-macosx-10.13.0-x86_64.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/matplotlib-3.3.0-py3.9-macosx-10.13.0-x86_64.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/GDAL-3.3.2-py3.9-macosx-10.13.0-x86_64.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/Pillow-7.2.0-py3.9-macosx-10.13.0-x86_64.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/patsy-0.5.1-py3.9.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/netCDF4-1.5.4-py3.9-macosx-10.13.0-x86_64.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/pandas-1.3.3-py3.9-macosx-10.13.0-x86_64.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/geopandas-0.8.1-py3.9.egg
/Applications/QGIS-LTR.app/Contents/MacOS/lib/python3.9/site-packages/numba-0.50.1-py3.9-macosx-10.13.0-x86_64.egg
/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python
/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/site-packages
/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin
/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/qgispluginsupport/qps/pyqtgraph
/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/site-packages
/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/apps
/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/coreapps
/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/enmapboxplugin/enmapbox/eo4qapps
.
/Users/lgzdl/Library/Application Support/gis4wrf/python39/lib/python3.9/site-packages
/Users/lgzdl/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/cgl_tools/extlibs

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.