stac-utils / qgis-stac-plugin Goto Github PK

For the moment, the GitHub page index is redirecting to 404.html
See the branch gh-pages

Add a list of open STAC APIs in the plugin as pre-populated connections. One example of the open STAC APIs is https://earth-search.aws.element84.com/v0

After search is complete, add the ability to download item assets and loading the item footprint into QGIS.
The current search item result widget provides buttons to load footprint and a combo box for downloading assets, we need to add functionality in the plugin to use the buttons to let user download assets and load the STAC items footprints.

For each connection, add a tool button to search and show the available conformance classes. This will enable knowing what operations are supported by the STAC API before using the connection in the plugin.
See https://github.com/radiantearth/stac-api-spec/blob/master/overview.md#conformance-classes for more information about conformance

Add scripts to enable running plugin tests locally on Windows and macOS operating systems.

The scripts should use the same approach used in the plugins linux bash script here https://github.com/stac-utils/qgis-stac-plugin/blob/main/run-tests.sh, which uses docker images, a test_suite module and QGIS testing environment to run the plugin tests.

After adding the scripts, add a section on the README file to explain how to use them.

When adding the item footprint, it appears that the geometry is added as a new layer. The item object itself is a valid GeoJSON object, and has valuable metadata in its properties field. Is it possible to just use the Item GeoJSON directly, and not create a new geometry so that the user can access the properties through QGIS?

I'm assuming this isn't the case currently, because there were not properties on the added footprints:

Currently the plugins test running on the Github actions are executed on a linux environment. We need to explore the possibility of running the tests in other OS environments eg.Windows and macOS.

This will enable the plugin tests to be checked in multiple environments in one go.

see https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners

Provide feedback and suggestions on the way forward for this task.

Provide settings and procedure for the plugin to be able to load and save connections details from and to files.

I'm finding that on my laptop (which has a relatively small screen), the main dialog window opens in such a way that it doesn't fit on the screen properly. The 'search' button at the bottom is off the bottom of the screen, and therefore can't be clicked. I've tried resizing the dialog, but that doesn't seem to work - presumably because there are too many controls to fit in a smaller dialog.

I've got two ideas for how to deal with this:

1. The various filtering options could start in their 'collapsed' state, so that they don't take up as much space.
2. The whole contents of the dialog could be put in a scrollable view, so that even if it takes up too much space, the user can scroll the dialog and eventually get to the end.

When loading the plugin for the first time, make sure we set the planetary computer as a default connection. The API is found here https://planetarycomputer.microsoft.com/api/stac/v1

When installing or upgrading the plugin in Windows OS with QGIS 3.16.14 an error occurs indicating there is a missing module called
typing_extensions

Couldn't load plugin 'qgis_stac' due to an error when calling its classFactory() method 

ModuleNotFoundError: No module named 'typing_extensions' 
Traceback (most recent call last):
  File "C:/PROGRA~1/QGIS3~1.16/apps/qgis-ltr/./python\qgis\utils.py", line 334, in _startPlugin
    plugins[packageName] = package.classFactory(iface)
  File "C:/Users/USER/AppData/Roaming/QGIS/QGIS3\profiles\yu/python/plugins\qgis_stac\__init__.py", line 41, in classFactory
    from .main import QgisStac
  File "C:/PROGRA~1/QGIS3~1.16/apps/qgis-ltr/./python\qgis\utils.py", line 793, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:/Users/USER/AppData/Roaming/QGIS/QGIS3\profiles\yu/python/plugins\qgis_stac\main.py", line 23, in 
    from .gui.qgis_stac_widget import QgisStacWidget
  File "C:/PROGRA~1/QGIS3~1.16/apps/qgis-ltr/./python\qgis\utils.py", line 793, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:/Users/USER/AppData/Roaming/QGIS/QGIS3\profiles\yu/python/plugins\qgis_stac\gui\qgis_stac_widget.py", line 21, in 
    from ..api.client import Client
  File "C:/PROGRA~1/QGIS3~1.16/apps/qgis-ltr/./python\qgis\utils.py", line 793, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:/Users/USER/AppData/Roaming/QGIS/QGIS3\profiles\yu/python/plugins\qgis_stac\api\client.py", line 3, in 
    from .base import BaseClient
  File "C:/PROGRA~1/QGIS3~1.16/apps/qgis-ltr/./python\qgis\utils.py", line 793, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:/Users/USER/AppData/Roaming/QGIS/QGIS3\profiles\yu/python/plugins\qgis_stac\api\base.py", line 19, in 
    from .network import ContentFetcherTask
  File "C:/PROGRA~1/QGIS3~1.16/apps/qgis-ltr/./python\qgis\utils.py", line 793, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:/Users/USER/AppData/Roaming/QGIS/QGIS3\profiles\yu/python/plugins\qgis_stac\api\network.py", line 20, in 
    from pystac_client import Client
  File "C:/PROGRA~1/QGIS3~1.16/apps/qgis-ltr/./python\qgis\utils.py", line 793, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\Users\USER\AppData\Roaming\QGIS\QGIS3\profiles\yu\python\plugins\qgis_stac\lib\pystac_client\__init__.py", line 4, in 
    from pystac_client.item_search import ItemSearch
  File "C:/PROGRA~1/QGIS3~1.16/apps/qgis-ltr/./python\qgis\utils.py", line 793, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\Users\USER\AppData\Roaming\QGIS\QGIS3\profiles\yu\python\plugins\qgis_stac\lib\pystac_client\item_search.py", line 11, in 
    from pystac import Collection, Item, ItemCollection
  File "C:/PROGRA~1/QGIS3~1.16/apps/qgis-ltr/./python\qgis\utils.py", line 793, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\Users\USER\AppData\Roaming\QGIS\QGIS3\profiles\yu\python\plugins\qgis_stac\lib\pystac\__init__.py", line 65, in 
    from pystac.collection import (
  File "C:/PROGRA~1/QGIS3~1.16/apps/qgis-ltr/./python\qgis\utils.py", line 793, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\Users\USER\AppData\Roaming\QGIS\QGIS3\profiles\yu\python\plugins\qgis_stac\lib\pystac\collection.py", line 32, in 
    from pystac.summaries import Summaries
  File "C:/PROGRA~1/QGIS3~1.16/apps/qgis-ltr/./python\qgis\utils.py", line 793, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\Users\USER\AppData\Roaming\QGIS\QGIS3\profiles\yu\python\plugins\qgis_stac\lib\pystac\summaries.py", line 24, in 
    from typing_extensions import Protocol
  File "C:/PROGRA~1/QGIS3~1.16/apps/qgis-ltr/./python\qgis\utils.py", line 793, in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'typing_extensions'


Python version: 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] 
QGIS version: 3.16.14-Hannover Hannover, 4279198632 

Python Path:
C:/PROGRA~1/QGIS3~1.16/apps/qgis-ltr/./python
C:/Users/USER/AppData/Roaming/QGIS/QGIS3\profiles\yu/python
C:/Users/USER/AppData/Roaming/QGIS/QGIS3\profiles\yu/python/plugins
C:/PROGRA~1/QGIS3~1.16/apps/qgis-ltr/./python/plugins
C:\PROGRA~1\QGIS3~1.16\apps\Python37
C:\PROGRA~1\QGIS3~1.16\apps\Python37\Scripts
C:\Program Files\QGIS 3.16\bin\python37.zip
C:\PROGRA~1\QGIS3~1.16\apps\Python37\DLLs
C:\PROGRA~1\QGIS3~1.16\apps\Python37\lib
C:\Program Files\QGIS 3.16\bin
C:\PROGRA~1\QGIS3~1.16\apps\Python37\lib\site-packages
C:\PROGRA~1\QGIS3~1.16\apps\Python37\lib\site-packages\win32
C:\PROGRA~1\QGIS3~1.16\apps\Python37\lib\site-packages\win32\lib
C:\PROGRA~1\QGIS3~1.16\apps\Python37\lib\site-packages\Pythonwin
C:/Users/USER/AppData/Roaming/QGIS/QGIS3\profiles\yu/python
C:\Users\USER\AppData\Roaming\QGIS\QGIS3\profiles\yu\python\plugins\qgis_stac\lib

see screenshot of the error below

Initially reported by @mmcfarland

Investigate previous QGIS STAC plugin from Radiant Earth to see if code contains useful items for this project

Firstly: this is a great plugin, thank you. I've wanted a QGIS plugin for STAC for ages, and this is absolutely brilliant! Now on to the actual issue:

The version of the pystac library that is included with this plugin is slightly out of date (understandably), but that means it doesn't include this bugfix. Without that PR, pystac fails to list the collections for any of my STAC API servers.

I'll submit a PR shortly to upgrade the included pystac.

The returned searched items in the plugin should be paginated to support browsing STAC APIs with large datasets.
See https://github.com/radiantearth/stac-api-spec/blob/master/item-search/examples.md#paging-examples.

Add search implementation for the STAC API items and collections in the plugin.
This includes using the extent, temporal and other provided fields in UI to filter the available STAC Catalogs. see https://github.com/radiantearth/stac-api-spec/tree/master/fragments/filter for filtering docs.
For reference, the items search parameters table https://github.com/radiantearth/stac-api-spec/tree/master/item-search#query-parameter-table

Add common metadata and extension properties as filters in the search tab.

When I try to activate the pluggin I get this error:

Do I need a more recent version of QGIS?

HI!
This seems to be a perfect plugin for QGIS users in Sweden. The Swedish NLS are shifting their access to API in favor of downloads and wms.
I have a working QGIS API connection to the Swedish NLS geodata storage. I have all credentials and can access the API as a standard WFS/OGC API connection within QGIS. However, trying the same access information and my existing OAuth2 auth configuration in the qgis-stac-plugin, I get the errors below. Where can I put in the header they are asking for?
NOTE: The sample API connections provided with the plugin are working, and I can add assets and stuff. The problem seems to be in is in the authorisation part.

2022-02-12T08:59:51     INFO    Testing connection...
2022-02-12T08:59:51     CRITICAL    Connection is not a valid STAC API
2022-02-12T08:59:59     INFO    Getting API conformance classes...
2022-02-12T08:59:59     WARNING    Problem in fetching content for https://api.lantmateriet.se/distribution/geodatakatalog/sokning/v1/detaljplan/v2.Error details, 900902Missing CredentialsRequired OAuth credentials not provided. Make sure your API invocation call has a header: 'Authorization : Bearer ACCESS_TOKEN'
2022-02-12T09:00:13     INFO    Fetching collections...
2022-02-12T09:00:13     CRITICAL    Problem in fetching content for https://api.lantmateriet.se/distribution/geodatakatalog/sokning/v1/detaljplan/v2.Error details, 900902Missing CredentialsRequired OAuth credentials not provided. Make sure your API invocation call has a header: 'Authorization : Bearer ACCESS_TOKEN'

When trying to load COG layers via QgsTask, QGIS application doesn't respond or crashes.

Screenshot

This error is found on draft branch found here https://github.com/Samweli/qgis-stac-plugin/tree/add_layers

Implement support for downloading STAC item assets via the download assets combo box that is available in the search result item widget.

Add basic framework on how the main functionality for browsing the STAC APIs will be implemented, this includes defining the models, api client logic and the use of the third party libraries.

It would be useful to also read static STAC catalogs.

Initially I'd say this would work by checking if the URL is a JSON document, then loading it and disabling the UI portions related to search, filtering, and pagination.

Rolling list of UI issues, suggestions, or considerations.

• Some assets may not have a title attribute. When they don't, the plugin list them in the asset dropdown as an empty string, which looks broken (or is at least unhelpful to the user). To account for this, the dropdown should use the asset Dict key as a suitable alternative to the title (e.g., SR_B1). An example can be found when viewing items on the Esri 10m Land Cover dataset.

Add a button to the Results tab which will add footprints for all items to QGIS. Ideally these would come as one layer with multiple features in that layer - but that may be a bit more tricky.

Add settings manager in the plugin to handle user, UI and global plugin settings management. The manager should be based on the QgsSettings https://qgis.org/pyqgis/3.2/core/other/QgsSettings.html.

Create a plugin message bar that will be responsible for providing notifications about the progress of plugin operations to user.
The progress bar should not distort the UI when it is shown.

Currently when adding a new connection, the connection is not persisted hence not shown in the connections list combo box.
Might be related to #16
screenshot

Due to recent plugin updates the results window no more uses QTreeView due to crashes that were caused by the delegate used by the view when painting result item widgets, now we are using QScrollArea as main view for the widgets..

QTreeView support items filtering and sorting out of the box using QSortFilterProxyModel but QScrollArea doesn't, so we need to implement these functionality inside the plugin.

We need to change the connection dialog window title to Edit Connection when opened in a edit mode and when fetching conformance classes we should disable all other window UI components.

Image showing a connection dialog showing Create New Connection title when opened in edit mode

Screenshot of connection dialog not disabling UI components when fetching

Add project skeleton for the initial plugin development, skeleton can be constructed from a template provided by plugin builder https://plugins.qgis.org/plugins/pluginbuilder3/.

Once the user have created a new connection we need to save the connection credential including the authentication configurations into the QGIS settings.
The plugin settings manager added in #13 is responsible for this operation.
Also we need to make sure the connection widget in the "Search" tab respond accordingly to the create, edit and delete actions on the connections.
The expected behavior is when a new connection is added, the connection should be active and selected from the connections list, when connection is deleted the last added connection should be active and selected.

Lets implement and incorporate a draft version of the plugin UI components.
The design of the UI should use the proposed mockups, see below
Search tab

Connection dialog

Search results tab

Setup infrastructure for testing in the plugin, the test suite should cover plugin functional features and include a mockup stac-api server for testing plugin functionality against stac-api spec.
The setup should enable running tests against QGIS LTR, QGIS nightly and QGIS 3.22 versions. See https://github.com/qgis/QGIS/tree/master/.docker/qgis_resources/test_runner for reference in running plugin test inside QGIS application.

Add support for the plugin to be able to store operations information and errors into local log file and remote log files.
For logs submitted remotely, add option for user to be able to enable/disable the feature.
Use python logging package https://docs.python.org/3/library/logging.html and QgsMessageLog class to make sure the logs appear in QGIS log messages panel, after they have been logged.

The planetary computer STAC API uses SAS token for access of some assets see https://planetarycomputer.microsoft.com/docs/concepts/sas/.
Add support in the plugin for user to be able to use the authentication api to access assets in the planetary computer API.

When searching for the first after installing the plugin an AttributeError happens

An error has occurred while executing Python code: 

AttributeError: 'NoneType' object has no attribute 'get_items' 
Traceback (most recent call last):
  File "/home/samweli/.local/share/QGIS/QGIS3/profiles/stac_install/python/plugins/qgis_stac/gui/qgis_stac_widget.py", line 227, in search_items
    self.api_client.get_items(
AttributeError: 'NoneType' object has no attribute 'get_items'


Python version: 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0] 
QGIS version: 3.22.0-Białowieża Białowieża, d9022691f1 

Python Path:
/usr/share/qgis/python
/home/samweli/.local/share/QGIS/QGIS3/profiles/stac_install/python
/home/samweli/.local/share/QGIS/QGIS3/profiles/stac_install/python/plugins
/usr/share/qgis/python/plugins
/usr/lib/python38.zip
/usr/lib/python3.8
/usr/lib/python3.8/lib-dynload
/home/samweli/.local/lib/python3.8/site-packages
/usr/local/lib/python3.8/dist-packages
/usr/lib/python3/dist-packages
/home/samweli/.local/share/QGIS/QGIS3/profiles/stac_install/python
/home/samweli/.local/share/QGIS/QGIS3/profiles/stac_install/python/plugins/qgis_stac/lib

screenshot

Add a form box in the search tab that will allow using advance queries when searching for items in the STAC API.
The information about filters that can be used is found here https://github.com/radiantearth/stac-api-spec/tree/master/item-search#filter.
The form box tooltip should notify the user that the used filter language will only work if it is supported by the connection STAC API.
STAC API filter fragment https://github.com/radiantearth/stac-api-spec/tree/master/fragments/filter#stac-api---filter-fragment

When displaying thumbnail for the STAC item, we should use item asset with role overview for thumbnail image if thumbnail is not available for the item. See https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#asset-roles for the available asset roles.
The asset with role overview usually contain possibly larger view than the thumbnail of the Item, we need to append height/width params to the URL to get a right size for thumbnail purposes.

Really useful plugin! Thanks!

The View Asset window, once opened, is not able to be closed (there is no button to close the window), which then blocks returning back to the stac browser window or even main map screen. I am forced to kill qgis via the terminal.

OS: Linux Mint 20.2
QGIS: 3.22.3

Using the available Add footprint button, implement support for loading items footprints inside QGIS as map layers.

After the item search is complete, we need to load and show the found results into the results tab. The results list should be a list of items in form of search result widgets. Also we need to add ability to filter and sort the results in UI see https://doc.qt.io/qt-5/qsortfilterproxymodel.html, implement QgsTasks for handling fetching, displaying an item thumbnail and adding items assets and footprints as layers inside QGIS.

When loading thumbnail images into the search result item widgets, thumbnail do not load properly or sometimes QGIS crashes.

Screenshot

Work branch https://github.com/Samweli/qgis-stac-plugin/tree/fix_thumbnail

The search filters values used when searching for items should be stored and loaded when plugin is opened the next time.
Using the plugin settings_manager, implement the store_search_filters and get_search_filters functions then use them in the main plugin widget to save and retrieve the filters.

The design for the result tab will include using a dedicated search result widget for showing search results, the widget will contain details and actions buttons for the result items.
We need to implement and add basic GUI functionality for the widget.
Below is the initial suggested mockup for the widget.

Hi,

Thank you for this plugin, it is an important contribution to the infrastructure around STAC.

I have tried the plugin against our STAC API test server and trying to load collections from it will crash QGIS completely. Maybe something is wrong with our service, but the plugin shouldnt crash QGIS - even with a malfunctioning API, I think.

We are not too keen on giving wide access to the service, but we will be happy to share the URL with the plugin team. Just write me at asger @ septima .dk

We noticed an issue preventing the interoperability with implementations that present different view of the data based on the Accept header. For example the GeoServer implementation uses the same endpoint for presenting an HTML view but also the STAC representations.

This could be easily fixed by including the Accept: application/geojson in all HTTP requests.

P.S.
One such endpoint is: https://registry.sage.uvt.ro/geoserver/ogc/stac

Implement logic to handle when edit and remove buttons for connections details have been clicked.
The remove button should prompt user to confirm about the action before removing the connection.

Implement a new dialog that will show collection detail view, with relevant collection attributes
(description, license, etc). Also include a link with rel "describedby" for external details.

Currently the plugin only highlight the collection properties in collection group box, see below screenshot
Plugin collection group box, showing a list of collections from Microsoft planetary computer catalog (https://planetarycomputer.microsoft.com/api/stac/v1)

We need a dedicated dialog that will display more information about a particular collection, see https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#collection-fields for a list of properties that a collection can contain.

if there are no settings.value("created_date") the plugin can't get Connection Setting