geopython / demo.pygeoapi.io Goto Github PK

Demo setup for https://demo.pygeoapi.io

License: MIT License

Io 0.67% Shell 47.42% Dockerfile 18.30% Python 10.47% JavaScript 3.04% HTML 20.09%

demo.pygeoapi.io's Introduction

geopython

Vanity package for geopython projects

pip install geopython

>>> import geopython

demo.pygeoapi.io's People

Contributors

Stargazers

Watchers

Forkers

justb4 pvgenuchten philopi ksonda jvanulde edwardpmorris isabella232 inspiregeo stjordanis matthiasdaues pascallike doublebyte1 chriss-0x01

demo.pygeoapi.io's Issues

Some Collections in pygeoapi master and stable endpoints give error on queryables

Problem

On the demo server, log in and execute docker logs --follow pygeoapi_stable. This gives errors like:

[2024-03-27T14:23:58Z] {/usr/local/lib/python3.10/dist-packages/flask/app.py:828} ERROR - Exception on /collections/mapserver_world_map/queryables [GET]
Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/util.py", line 542, in get_provider_by_type
    p = (next(d for i, d in enumerate(providers)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/api.py", line 1508, in get_collection_queryables
    p = load_plugin('provider', get_provider_by_type(
  File "/pygeoapi/pygeoapi/util.py", line 545, in get_provider_by_type
    raise ProviderTypeError('Invalid provider type requested')
pygeoapi.provider.base.ProviderTypeError: Invalid provider type requested

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/util.py", line 542, in get_provider_by_type
    p = (next(d for i, d in enumerate(providers)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/api.py", line 1513, in get_collection_queryables
    p = load_plugin('provider', get_provider_by_type(
  File "/pygeoapi/pygeoapi/util.py", line 545, in get_provider_by_type
    raise ProviderTypeError('Invalid provider type requested')
pygeoapi.provider.base.ProviderTypeError: Invalid provider type requested

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pygeoapi/pygeoapi/util.py", line 542, in get_provider_by_type
    p = (next(d for i, d in enumerate(providers)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.10/dist-packages/flask_cors/extension.py", line 176, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  File "/pygeoapi/pygeoapi/flask_app.py", line 213, in collection_queryables
    return get_response(api_.get_collection_queryables(request, collection_id))
  File "/pygeoapi/pygeoapi/api.py", line 228, in inner
    headers, status, content = func(*args, **kwargs)
  File "/pygeoapi/pygeoapi/api.py", line 210, in inner
    return func(cls, req_out, *args[2:])
  File "/pygeoapi/pygeoapi/linked_data.py", line 60, in inner
    return func(*args, **kwargs)
  File "/pygeoapi/pygeoapi/api.py", line 1517, in get_collection_queryables
    p = load_plugin('provider', get_provider_by_type(
  File "/pygeoapi/pygeoapi/util.py", line 545, in get_provider_by_type
    raise ProviderTypeError('Invalid provider type requested')
pygeoapi.provider.base.ProviderTypeError: Invalid provider type requested

Analysis

Cannot reproduce these errors the https://demo.pygeoapi.io/stable/collections/mapserver_world_map seems to work.
Some other remote client seems to trigger the errors, also for some other collections, not all, e.g. https://demo.pygeoapi.io/stable/collections/dutch_windmills/queryables works ok.
Though taking from the error this gives an Exception: https://demo.pygeoapi.io/stable/collections/collections/mapserver_world_map/queryables
Also on master endpoint: https://demo.pygeoapi.io/master/collections/mapserver_world_map/queryables
As #52 has copied the config from the master endpoint, is there a facility (queryables?) not yet present in stable pygeoapi v0.16.0?

Migrate demo server setup to GitOps

The current setup to maintain and deploy the various services is semi-automated: Ansible is used, but with full GitOps, coupling GitHub, GH Workflows with Ansible, we get several advantages:

as pygeoapi Docker builds are moved to GH Workflows, will be easier to integrate (now HTTP-callback from DockerHub)
any other service like the "Home" page will be automagically deployed on GH push
repo is well-prepared to later move to OSGeo SAC infrastructure

As example of a fully fledged GitOps pygeoapi deployment the Geonovum OGC API Testbed project can serve as example:

Add Dutch COVID-19 layers to demo

In https://demo.pygeoapi.io/covid-19 global COVID-19 is shown as directly proxied from official sources like the JHU.

Via this issue cases/layers are added specific for The Netherlands.

Automate redeployments with GitHub Actions

Changes to the pygeoapi GitHub repo are automatically propagated via DockerHub and a DockerHub listener here.

But changes to the demo.pygeoapi.io repo like config or data changes here are not auto-deployed.
Manual action via Ansible is required like:

ansible-playbook -vv deploy.yml -i hosts/demo.pygeoapi.io --tags update,pygeoapi_master etc

Also one needs SSH access.

To have full CI deployment GitHub Actions need to be developed such that anyone with commit rights to the demo repo here can redeploy a service.

Hint: there are quite some GH Actions Ansible modules available, the commands are simple enough, SSH-keying may be the biggest challenge, docs:

https://misfra.me/2019/10/using-ansible-with-github-actions/ - seems simplest solution
extensive article on Medium

service is down.

Could you please take a look the service link below, which is down? Thanks.

https://demo.pygeoapi.io/covid-19

WFS Service GeoSolutions stale

See e.g. https://demo.pygeoapi.io/master/collections/unesco_pois_italy/items
Used for OGR WFS Proxy demo in latest. See also this ticket:

Dear Just van den Broecke,

We would like to acknowledge that we have received your request and a ticket has been created with Ticket ID - 48684.
A support representative will be reviewing your request and will send you a personal response within the agreed SLA.


To view the status of the ticket or add comments, please visit
https://geosolutions.freshdesk.com/support/tickets/48684 (this links requires login).


A read-only version of the ticket is visible here

https://geosolutions.freshdesk.com/public/tickets/556ff83878694be1aac192f0f9a475079ffe4604d91041302c348f04b1e3e2d2


Thank you for your patience.

Sincerely,
GeoSolutions Group Support Team

In relation to geopython/pygeoapi#325, I would like to setup an instance at https://demo.pygeoapi.io/cite which provides CITE specific data, configuration and endpoint against pygeoapi master. The instance needs to have a very small and basic Elasticsearch backend (with a single index of 50 records) instance of to expose the full functionality needed for CITE. Basic workflow on a commit would be:

stand up ES instance
load a single index with 50 records
generate OpenAPI document
start server

Errors on queryables requests

See e.g. https://demo.pygeoapi.io/master/collections/dutch_windmills/queryables
gives:
Internal Server Error The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

for all endpoints using master version.

Possibly need config updates (or in Providers?)

In some cases also error output is shown, like on
https://demo.pygeoapi.io/covid-19/collections/cases/queryables

[2020-04-25T10:43:45Z] {/usr/local/lib/python3.8/dist-packages/flask/app.py:1891} ERROR - Exception on /collections/cases/queryables [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/pygeoapi/pygeoapi/flask_app.py", line 181, in get_collection_queryables
    headers, status_code, content = api_.get_collection_queryables(
  File "/pygeoapi/pygeoapi/api.py", line 88, in inner
    return func(cls, headers_, format_, *args, **kwargs)
  File "/pygeoapi/pygeoapi/linked_data.py", line 54, in inner
    return func(*args, **kwargs)
  File "/pygeoapi/pygeoapi/api.py", line 518, in get_collection_queryables
    'type': v['type']
TypeError: string indices must be integers

Fix config for changes in pygeoapi PR geopython/pygeoapi#244

geopython/pygeoapi#244 introduced changes in config. These need to be reflected in the master demo.

Now error on get collections:

[2019-09-26T06:16:20Z] {/usr/local/lib/python3.7/dist-packages/flask/app.py:1891} ERROR - Exception on /collections [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.7/dist-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/pygeoapi/pygeoapi/flask_app.py", line 129, in describe_collections
    request.headers, request.args, name)
  File "/pygeoapi/pygeoapi/api.py", line 78, in inner
    return func(cls, headers_, format_, *args, **kwargs)
  File "/pygeoapi/pygeoapi/api.py", line 300, in describe_collections
    'crs': v['extents']['spatial']['crs']
KeyError: 'crs'

upgrade to bionic

cc @justb4

Should we consider moving the demo VM setups to 18.04?

add OATiles demo

As discussed at the May 2021 OGC API Virtual Code Sprint, add OATiles example. cc @francbartoli

add a OACov demo

Prepare demo with data for OACOV using rasterio/xarray. Example conf and data

@tomkralidis Maybe I can prepare some dataset from soilgrids project

pygeoapi_master does not start after upgrade to 0.15.0

docker logs --follow pygeoapi_master
START /entrypoint.sh
Trying to generate openapi.yml
Traceback (most recent call last):
  File "/usr/local/bin/pygeoapi", line 33, in <module>
    sys.exit(load_entry_point('pygeoapi', 'console_scripts', 'pygeoapi')())
  File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/pygeoapi/pygeoapi/openapi.py", line 1360, in generate
    content = generate_openapi_document(config_file, format_)
  File "/pygeoapi/pygeoapi/openapi.py", line 1335, in generate_openapi_document
    content = yaml.safe_dump(get_oas(s), default_flow_style=False)
  File "/pygeoapi/pygeoapi/openapi.py", line 1292, in get_oas
    return get_oas_30(cfg)
  File "/pygeoapi/pygeoapi/openapi.py", line 1132, in get_oas_30
    p = load_plugin('process', v['processor'])
  File "/pygeoapi/pygeoapi/plugin.py", line 113, in load_plugin
    class_ = getattr(module, classname)
AttributeError: module 'pygeometa.pygeoapi_plugin' has no attribute 'PygeometaMetadataSchemasProcessor'. Did you mean: 'PygeometaMetadataGenerateProcessor'?
ERROR: openapi.yml could not be generated ERROR

Analysis

Almost sure this is due to that pygeometa v0.14.0 is installed, but pygeometa.pygeoapi_plugin.PygeometaMetadataSchemasProcessor is not yet in that/any version, only in main branch.
So commenting out in local.config.yml will fix.

Bring configs uptodate with OAPIF Part 2 (CRS support)

Now that pygeoapi supports Part 2 - CRS support, it makes sense to upgrade various configs on the demo server to showcase. Although defaults (WGS84 with lon,lat axis ordering) apply, it would be good to show support and especially gear up for CITE tests.

Basically for each 'feature' provider the crs and storageCRS config params need to be added, for example:

        providers:
            - type: feature
              name: Elasticsearch
              data: http://elasticsearch_cite:9200/canada-hydat-daily-mean-02hc003
              id_field: IDENTIFIER
              time_field: DATE
# NEW
              crs:
                - http://www.opengis.net/def/crs/OGC/1.3/CRS84
                - http://www.opengis.net/def/crs/EPSG/0/4326
                - http://www.opengis.net/def/crs/EPSG/0/3348
              storage_crs: http://www.opengis.net/def/crs/OGC/1.3/CRS84

http://www.opengis.net/def/crs/OGC/1.3/CRS84 basically means WGS84 with lon,lat axis ordering, the default.

Update endpoints running master (latest Docker Image).

/cite endpoint
/master endpoint, also remove all source_srs and target_srs from OGR Providers. source_srs should be replaced with corresponding storage_crs.
/covid endpoint, though may be obsolete..

Add dataset for OGC API Tiles

It would be great to add an example dataset, served with OGC API (vector) Tiles. We could leverage ElastiSearch as data provider for that.

An example can be found here:
https://github.com/emotional-cities/openapi-sdi/blob/master/pygeoapi/docker.config.local.yml#L114

pygeoapi config YAMLs: fix temporal and bbox format issues

needs to be fixed for master and stable pygeoapi YAML configs on demo site:

temporal "NULL" values: geopython/pygeoapi#237
bbox array format: geopython/pygeoapi#239

Add Italy COVID-19 layers to demo

In https://demo.pygeoapi.io/covid-19 global COVID-19 is shown as directly proxied from official sources like the JHU.

Via this issue cases/layers are added specific for the Italy.

Update config for stable release service endpoint

Problem

Latest stable https://demo.pygeoapi.io/stable gives some strange results. For example:

windmills dataset shows wrong location: https://demo.pygeoapi.io/stable/collections/dutch_windmills/items/Molens.1
maybe others, probably all the layers with other CRS-es than EPSG:4326

Analysis

local.config.yml still has the "old" pre 0.15.0 content:

Part 3 CRS-support requires config changes storageCRS etc. Many layers are in Dutch EPSG:28992
More, tiles?
Should we just copy local.config.yml and data dir from the master endpoint (except config , title etc) ? Let's just try!

bbox data extent wrong or reversed

Many Collections in all demo endpoints show a data extent that is

too wide, showing world coverage i.s.o. small area
axis-order reversed, mainly Dutch data Collections like Windmills

This can be fixed withiin each endpoint config yaml.

page can not be crawled due to robots.txt

We have set up a ggl crawling on demo.pygeoapi.io to research crawler behaviour on pygeoapi. First results are available, but it puzzels me a bit.

Ggl generally crawls pygeoapi pages in a correct way. One can indeed find demo pygeoapi results at for example https://www.google.com/search?q=site%3Ademo.pygeoapi.io. however no results yet at https://toolbox.google.com/datasetsearch/search?query=site%3Ademo.pygeoapi.io

A weird thing is that when doing 'live test' (a feature on ggl search console) on this url https://demo.pygeoapi.io/master/collections/lakes i get this error: "url not available to google, blocked by robots.txt"

However https://demo.pygeoapi.io/master/collections/lakes?f=html runs fine in 'live test'. This makes me wonder, does the 'live-test' crawler use the proper accept header?

Another thing to improve is the fact that https://demo.pygeoapi.io/robots.txt does not return a proper robots.txt file, but in stead a custom file-not-found page (with http status 200!)

let me now if you have any ideas

add a basic stac configuration to the demo instance

it is usefull to have a small configuration of a stac dataset to demonstrate the wide range of usecases supported by pygeoapi, also usefull for developers te develop and/or check complience of/to stac clients

Fix failing Collections

Several Collections are failing on master latest and possibly in versions. At least:

Dutch Georef stations (backend WFS URL has changed): https://demo.pygeoapi.io/master/collections/dutch_georef_stations/items
"Utah" with deegree WFS works on https://demo.pygeoapi.io/master/collections/utah_city_locations but the OGR Provider gives errors on startup and feature fetching:
[2023-01-18T14:31:02Z] {/pygeoapi/pygeoapi/provider/ogr.py:811} ERROR - Error Number: 1, Type: Failure, Msg: Error returned by server : HTTP error code : 400 (0)
Large Lakes OGR GeoJSON when fetching any Item: https://demo.pygeoapi.io/master/collections/ogr_geojson_lakes/items/0

[2023-01-18T14:33:17Z] {/usr/lib/python3/dist-packages/flask/app.py:1890} ERROR - Exception on /collections/ogr_geojson_lakes/items/0 [GET]
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python3/dist-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/pygeoapi/pygeoapi/flask_app.py", line 219, in collection_items
    api_.get_collection_item(request, collection_id, item_id))
  File "/pygeoapi/pygeoapi/api.py", line 195, in inner
    headers, status, content = func(*args, **kwargs)
  File "/pygeoapi/pygeoapi/api.py", line 177, in inner
    return func(cls, req_out, *args[2:])
  File "/pygeoapi/pygeoapi/api.py", line 2216, in get_collection_item
    content = render_j2_template(self.config,
  File "/pygeoapi/pygeoapi/util.py", line 378, in render_j2_template
    return template.render(config=l10n.translate_struct(config, locale_, True),
  File "/usr/local/lib/python3.8/dist-packages/jinja2/environment.py", line 1291, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.8/dist-packages/jinja2/environment.py", line 925, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/pygeoapi/pygeoapi/templates/collections/items/item.html", line 2, in top-level template code
    {% set ptitle = data['properties'][data['title_field']] or "Item " + data['id'] %}
TypeError: can only concatenate str (not "int") to str

Latter 3 errors are a real pygeoapi issue as they don't appear on the "stable" instance e.g.

add raw sources of Dutch metadata

As discussed at the OGC Virtual Sprint, we should have the raw XML sources of the Dutch metadata collection in https://github.com/geopython/demo.pygeoapi.io/tree/master/services/pygeoapi_master/data/nationaalgeoregister.nl, along with possibly a README.md with an overview/link to authoritative source(s).

Upgrade pygeoapi stable service to 0.8.0

Now that pygeoapi 0.8.0 is out pygeoapi stable needs upgrade from 0.7.0:

upgrade docker-compose file
update/upgrade config file for format changes
roll out on demo.pygeoapi.io

geopython / demo.pygeoapi.io Goto Github PK

demo.pygeoapi.io's Introduction

geopython

demo.pygeoapi.io's People

Contributors

Stargazers

Watchers

Forkers

demo.pygeoapi.io's Issues

Problem

Analysis

Analysis

Problem

Analysis

Recommend Projects

Recommend Topics

Recommend Org