bird-house / emu Goto Github PK

See: https://pypi.org/project/six/

Description

Install fails due to conda dependency errors ... see below.

Steps to Reproduce

See failed build on travis:

$ python setup.py install
Traceback (most recent call last):
  File "setup.py", line 5, in <module>
    version = __import__('emu').__version__
  File "/home/travis/build/bird-house/emu/emu/__init__.py", line 1, in <module>
    from .wsgi import application
  File "/home/travis/build/bird-house/emu/emu/wsgi.py", line 2, in <module>
    from pywps.app.Service import Service
  File "/home/travis/miniconda/envs/emu/lib/python3.6/site-packages/pywps/__init__.py", line 10, in <module>
    from lxml.builder import ElementMaker
ImportError: libiconv.so.2: cannot open shared object file: No such file or directory

Additional Information

https://travis-ci.org/bird-house/emu

Description

It happens quite often that we have a set of processes with common input (and output) parameters. In WPS the process signature (inputs+outputs) is called a profile (I think). To avoid copy+paste of this process parameters one could use Python decorators or mixin classes to define a commonly used profile which can be adapted by each individual process.

Here are some examples how it could look like:

• notebooks: https://github.com/cehbrecht/jupyterlab-notebooks/tree/master/pywps-profiles
• Emu subset with ESGF-API: https://github.com/bird-house/emu/blob/esgfwps/emu/processes/wps_esgf_subset.py

Environment

• Emu version used, if any:
• Python version, if any:
• Operating System:

Steps to Reproduce

Additional Information

Description

Processes should provide the percent completed information so we can log their progress.

Description

... otherwise I get an exception like this:
OSError: [Errno 48] Address already in use

... and I loose the pid file and can' just kill the running service with make stop.

Environment

• Emu version used, if any: 0.8.0
• Python version, if any:
• Operating System:

Steps to Reproduce

$ make start
$ make status
$ make start
OSError: [Errno 48] Address already in use

Additional Information

latest pywps from conda (pywps-4.0.0-py27_36.) produces duplicate outputs, out.txt and out_dNeESE.txt for example:

2018-02-05 14:11:21,158] [INFO] Storing file output to /Users/pingu/birdhouse/var/lib/pywps/outputs/emu/0b37a440-0a76-11e8-966f-acde48001122/out.txt
2018-02-05 14:11:21,163] [INFO] Storing file output to /Users/pingu/birdhouse/var/lib/pywps/outputs/emu/0b37a440-0a76-11e8-966f-acde48001122/out_dNeESE.txt

Description

RTD supports (beta):
https://docs.readthedocs.io/en/latest/conda.html

Additional Information

See also #16.

... this is who the other GeoPython projects do it. Tests folder can then be kept outside of the package if not wanted.

gunicorn=19.8 does not like the passed header X-Forwarded-Protocol=https.

See ticket: bird-house/flyingpigeon#243

Description

This docs are silent about the usage of bind_host.

From reading the code the relevant parameter is hostname. This parameter is used in the pywps config and well be used in the wps response documents.

But one can use bind_host to set the IP address used to start the service … but this is not part of the pywps config. Bind_host is used in the docker container.

Describe what you were trying to get done or your feature request.

Environment

• Emu version used, if any: 0.9.0
• Python version, if any:
• Operating System:

Steps to Reproduce

Additional Information

Description

Current version is just using Conda and Werkzeug to start the WPS service for development and demo. The Sphinx docs needs to be updated.

See related ticket: bird-house/flyingpigeon#240

see https://github.com/Unidata/MetPy

Description

The current test suite has copied util methods form pywps, like get_output:

We should make the pywps test utils public available. We should also have a look at Get and Post test examples.

Common test tools should be part of pywps and not copied by each WPS bird.

Additional Information

See also PR #48.

... not really used here ... just to reduce the deps to get started ...

Need to use threaded mode in werkzeug config, threaded=True or processes=2:
http://werkzeug.pocoo.org/docs/0.14/serving/

Description

Use click command line instead of argparse.

Additional Information

bird-house/cookiecutter-birdhouse#8

Description

We have a lot of processes that return netCDF files. As of now, they are served as file URL. It would be useful to be able to serve them as well through DAP.

I suggest Emu includes an option to configure and launch a netCDF handler based on pydap to serve the outputs created by Emu. In production, we would use THREDDS, but I think it would be easier for testing to use pydap.

See geopython/pywps#513

Description

The latest pywps command line interface introduced in the watchdog PR is missing the daemon mode and has therefore only a start command but no stop/status commands. We like to have the pywps daemon mode back.

See comments in PR #95

Environment

• Emu version used, if any:
• Python version, if any:
• Operating System:

Steps to Reproduce

Additional Information

Description

• Add process including translation.
• Configure server

Description

The PyWPS demo is an app build with Flask:
https://github.com/geopython/pywps-flask

It is not necessary to use it, it would add another dependency ... but we could have a nice front-page like in the pywps-demo. We could make this decision optional in the cookiecutter.

Environment

Steps to Reproduce

Additional Information

... called by travis. See pywps how to do this.

Description

Use bumpversion to set version in docs/source/conf.py.

Description

Werkzeug can also start the WPS service with HTTPS. See Werkzeug documentation.

Environment

• Emu version used, if any:
• Python version, if any:
• Operating System:

Steps to Reproduce

Additional Information

Description

SupportedFormat for output is FORMATS.JSON but the actual content cannot be parsed by json.loads.

probably need to fix owslib/pywps to make this work.

... see Birdy example.

Description

Currently the poly_centroid process supports only GML/XML as input format. We could extend this for Well-known-txt (WKT), which is easier to type manually in demos.

Additional Information

See PR #45.

Description

Using Jinja template for pywps configuration to make it adaptable.

see:
https://github.com/blog/2111-issue-and-pull-request-templates

Description

I think it would be useful to have a public live emu server that can be used for testing. The server would have both anonymous access and known credentials (testuser, testpassword) so we can test clients.

Description

With the new deployment relying on Werkzeug we currently have no make stop to shutdown the service.

It could be implemented by remembering the PID and using kill or using a function provided by Werkzeug:

http://werkzeug.pocoo.org/docs/0.14/serving/#shutting-down-the-server

Description

Just return a very small basic netCDF dataset to test clients.

Description

handler method should be a staticmethod.
See pywps documentation.

Hi,

I am trying to build docker image for emu. At some point I get this:

Installing Anaconda ...
mkdir: cannot create directory '//anaconda': Permission denied
ERROR: Could not create directory: //anaconda
make: *** [anaconda] Error 1

I am not sure if this is problem in the Dockerfile, or something wrong with my docker installation.

Description

The Ansible pywps.cfg has changed to support the installation of multiple pywps instances:
https://github.com/bird-house/ansible-wps-playbook/blob/master/roles/pywps/templates/pywps.cfg

It is using now item which comes from using the Ansible with_items loop.

Steps to Reproduce

Additional Information

See ticket in Ansible playbook: bird-house/ansible-wps-playbook#9

Description

One of the processes nc_meta needs the netcdf4 library. Not a big deal currently in Emu ... but we could make this an optional dependency to have an example how to handle this.

Environment

• Emu version used, if any: 0.9
• Python version, if any:
• Operating System:

Steps to Reproduce

Additional Information

Description

When just running make test the log file is empty though it should contain some logs according to the default.cfg.

Environment

• Emu version used, if any: 0.9.1
• Python version, if any:
• Operating System:

Steps to Reproduce

Try the following:

$ pytest --log-cli-level=DEBUG

__init__.py                 15 DEBUG    setting core variables
_compat.py                  25 DEBUG    Python 3.x
configuration.py            69 INFO     loading configuration
configuration.py            75 DEBUG    setting default values
configuration.py           188 DEBUG    UNIX based environment
configuration.py           200 DEBUG    trying to estimate the default location
configuration.py           144 INFO     No configuration files loaded. Using default values

Looks like the pywps service in the tests does not load our default.cfg.

Additional Information

https://docs.pytest.org/en/latest/logging.html

see: https://stickler-ci.com/

to be used as the template basis.

Description

Add a dummy process whose identifier contains characters that are not allowed in python variables.
Would be useful to test owslib and birdy.

Description

Use Conda to build an environment that should be identical over time and platform:

https://conda.io/docs/user-guide/tasks/manage-environments.html#building-identical-conda-environments

Steps to Reproduce

Additional Information

and update travis:

http://conda.pydata.org/docs/travis.html

Description

To test birdy client.

https://pypi.python.org/pypi/pycodestyle

... it was using requirements/rtd.txt.

... with chaining ... for tutorial and demo.

Description

In the _handler, we have

        def words(f):
            for line in f:
                for word in wordre.findall(line.decode('UTF-8')):
                    yield word

which assumes line has a decode method, but the supported_format (TEXT) does not explicitly specify that the encoding is utf-8. So if the content is passed as an embedded string in the request, with no encoding information, the process fails.

Description

Using a Metalink file to store references to multiple output files.

 <?xml version="1.0" encoding="UTF-8"?>
 <metalink xmlns="urn:ietf:params:xml:ns:metalink">
   <published>2009-05-15T12:23:23Z</published>
   <file name="example.ext">
     <size>14471447</size>
     <identity>Example</identity>
     <version>1.0</version>
     <language>en</language>
     <description>
     A description of the example file for download.
     </description>
     <hash type="sha-256">3d6fece8033d146d8611eab4f032df738c8c1283620fd02a1f2bfec6e27d590d</hash>
     <url location="de" priority="1">ftp://ftp.example.com/example.ext</url>
     <url location="fr" priority="1">http://example.com/example.ext</url>
     <metaurl mediatype="torrent" priority="2">http://example.com/example.ext.torrent</metaurl>
   </file>
 </metalink>

In a few other birds, I've added a recipe for running tests on Jupyter Notebooks. This allows for users to evaluate whether the output of their notebooks is consistent on their environment.

This requires a few more libraries (jupyter, nbval) that could be added to the environment.yml or their requirements.txt (ideally, this would be installed via Anaconda). I would also add an example notebook that attempts to load emu as a smoke test.