idemsinternational / epicsa-climate-api Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 0.0 176 KB

API endpoints to process climate data for use in E-PICSA

Home Page: https://api.epicsa.idems.international

License: MIT License

Dockerfile 1.52% Python 96.89% Shell 0.52% R 1.07%

epicsa-climate-api's Introduction

E-PICSA Climate Api

API for accessing E-PICSA climate services

Build with FastAPI

Pre-Requisites

The api requires Python and R runtimes installed. It has been tested with the versions listed below

Python (3.11)
https://www.python.org/downloads
R and Rtools (4.2.3)

Windows
https://cran.r-project.org/bin/windows/base/old/4.2.3/
https://cran.r-project.org/bin/windows/Rtools/rtools42/rtools.html

Linux
https://cran.r-project.org/bin/linux/

Mac (untested)
https://cran.r-project.org/bin/macosx/

Relevant documentation should also be followed to ensure runtimes can be executed from PATH environment variable.

Configuration

Environment

Create an environment file from the sample. These will be populated using Pydantic

cp .env.sample .env

Python

The scripts below will create a python virtual environment, activate, install required python and R dependencies and start local server

=== "Windows (powershell)"

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install --upgrade -r requirements.txt
uvicorn app.main:app --reload

=== "Linux (bash)"

python -m venv .venv
source .venv/bin/activate
pip install --upgrade -r requirements.txt
uvicorn app.main:app --reload

Once installed you will need to call R from an elevated shell to install dependencies

Windows

Rscript install_packages.R
Rscript install_packages_picsa.R

Linux

sudo Rscript install_packages.R
sudo Rscript install_packages_picsa.R

Authorization File

In order to run the package, you will need to add the following service account file to the main repository folder (i.e. the folder where this README.md file is stored):

service-account.json

Running locally

Once installed, subsequent server starts can skip installation steps

=== "Windows (powershell)"

.\.venv\Scripts\Activate.ps1
uvicorn app.main:app --reload

=== "Linux (bash)"

source .venv/bin/activate
uvicorn app.main:app --reload

The server will start at http://127.0.0.1:8000

Running Tests

pytest

Running locally (docker)

docker compose up --build

Deployment

This repo contains example workflow to build as a docker image and deploy to google cloud run. See action yaml for details

Troubleshooting

Pip won't install dependencies Depending on local versions of R and python (as well as operating system) there may be issues when installing certain packages. Recommend attempting install using the requirements_dev.txt file which pins exact versions of packages shown to be compatible with each other, i.e.

pip install --upgrade -r requirements_dev.txt

R Dependency incompatibility Ensure R installed a per prerequisites, verify version via rscript --version.

If facing issues with a specific package it may help to download RStudio, and using the packages tab to check what version of packages are installed and update any indicated within error logs.

Called endpoint method does not exist The library calls methods from various other git repos where code is hosted both in python and R. These are installed during initial setup, but will need reinstallation whenever new versions of the external repos exist.

Simply repeat the steps above to install dependencies from install_packages_picsa.R

Any other issues should be raised on GitHub

License

This project is licensed under the terms of the MIT license.

epicsa-climate-api's People

Contributors

epicsa-climate-api's Issues

Create endpoints to allow data to be passed in

Create the endpoints to allow station raw data or summarised station to be passed in.

IDEMSInternational/epicsawrap#51

Create endpoints for Extreme events

Create corresponding endpoints for IDEMSInternational/epicsawrap#52

In README.md file, 'fastapi run' command gives error

@chrismclarke The top-level README.md file contains the following Windows installation instructions:

    python -m venv .venv
    .\.venv\Scripts\Activate.ps1
    pip install -r requirements.txt
    fastapi run

The last line gives me an error.
Should it be something like uvicorn app.main:app --reload?
Or have I missed something?

FYI, the error was 'The term 'fastapi' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.'.

I would normally start to investigate the path, but I remembered that for Open CDMS we ran uvicorn so I thought I'd check with you first.

Create endpoints for forecasts

Create endpoints to return short-term and long-term forecasts.

Not fully defined where these are stored or how many there will be so at the moment.

First draft should be a place holder that shows public accessible PDF

Create end point to return definitions for a station

Create end point for definitions to be returned

IDEMSInternational/epicsawrap#48

update end points to return station metadata

Once work is completed in epicsawrap complete exiting end points

IDEMSInternational/epicsawrap#50

annual_rainfall_summaries - if dates aren't present they come back as {} not ""

This is an issue with rpy2 which looks like it has been resolved now. Will need to update in order to fix

Migrate all tests from epics-python

As epicsa python as now been included in the API the tests will need to be transferred see https://github.com/IDEMSInternational/epicsa-python/blob/main/tests/test_epicsa_python.py

Create end point to allow a new definitions file to be added

Create end point for: IDEMSInternational/epicsawrap#49

We may need to think about adding authorisation to this end point as it should only be accessed by climsoft and not public facing.

Bug(annual_rainfall_summaries): NA_character_ present in metadata

For some specific endpoint requests the metadata comes back with NA_character_ field which cannot be handled by the API.

E.g. using

{
  "country": "zm",
  "station_id": "16",
  "summaries": [
    "annual_rain",
    "start_rains",
    "end_rains",
    "end_season",
    "seasonal_rain",
    "seasonal_length"
  ]
}

Response metadata comes back as:

OrderedDict([('annual_rain', OrderedDict([('annual_rain', 'TRUE'), ('n_rain', 'TRUE'), ('na_rm', 'FALSE')])), ('start_rains', OrderedDict([('threshold', 1), ('s_start_doy', 120), ('start_day', 1), ('end_day', 366), ('total_rainfall', 'TRUE'), ('amount_rain', 25), ('over_days', 3), ('proportion', 'FALSE'), ('number_rain_days', 'FALSE'), ('dry_spell', 'TRUE'), ('spell_max_dry_days', 10), ('spell_interval', 21), ('dry_period', 'FALSE'), ('last_updated', '2022-11-25')])), ('end_rains', OrderedDict([('start_day', 121), ('end_day', 366), ('interval_length', 1), ('min_rainfall', 10), ('s_start_doy', 120)])), ('end_season', OrderedDict([('start_day', 121), ('end_day', 366), ('capacity', 100), ('water_balance_max', 60), ('evaporation', 'value'), ('evaporation_value', 5), ('s_start_doy', 120)])), ('seasonal_rain', OrderedDict([('seasonal_rain', 'TRUE'), ('end_type', 'rains'), ('total_rain', 'TRUE'), ('n_rain', 'TRUE'), ('na_rm', 'FALSE'), ('rain_day', 0.85)])), ('seasonal_length', OrderedDict([('end_type', 'rains')])), (NA_character_, OrderedDict([('start_day', 121), ('end_day', 366), ('interval_length', 1), ('min_rainfall', 10)]))])

see end of response for OrderedDict([('end_type', 'rains')])), (NA_character_, OrderedDict([('start_day', 121)....

[Bug] Server uptime check fails randomly

The server contains an uptime check which ensures a service-account.json file exists as a quality-control check. This seems to somewhat randomly pass/fail, even when the file is present (see video below).

This only appears to happen when running in google cloud (can't reproducde locally)

E-PICSA.Climate.API.-.Swagger.UI.webm

@router.get("/")
def get_status():
    """
    Check server up and authorized to access data
    """
    checkServiceAccount()
    # TODO - add test within epicsa wrapper to check if R installed and correct version
    return 'Server Up'


def checkServiceAccount():
    # check for service-account.json file relative to project root
    serviceJsonPath = path.abspath('./service-account.json')
    if path.exists(serviceJsonPath):
        return True
    else:
        raise HTTPException(status_code=401,detail="service-account.json missing from local files. Endpoints will not be able to access climate data")

crop_success_probabilities sample data

Need to add examples and tests for station test_1

What would be good examples for:
water_requirements
planting_length
planting_dates

@lilyclements @rdstern Are you able to give an example I could use?