dgilford / tcpypi Goto Github PK

tcpyPI, aka "pyPI": Tropical Cyclone Potential Intensity Calculations in Python

License: MIT License

Python 1.64% MATLAB 0.99% Jupyter Notebook 97.38%

climate climate-data weather-conditions hurricane-calculations potential-intensity hurricanes tropical-cyclone tropical-cyclone-variability

tcpypi's Introduction

tcpyPI: Potential Intensity Calculations in Python

tcpyPI, 'pyPI' for short, is a set of scripts and notebooks that compute and validate tropical cyclone (TC) potential intensity (PI) calculations in Python. It is a fully documented and improved port of the Bister and Emanuel 2002 algorithm (hereafter BE02) which was originally written in FORTRAN---and then MATLAB---by Prof. Kerry Emanuel (MIT). Kerry's original MATLAB code (pcmin.m) is found at:

http://texmex.mit.edu/pub/emanuel/TCMAX

The goals in developing and maintaining pyPI are to:

supply a freely available validated Python potential intensity calculator,
carefully document the BE02 algorithm and its Python implementation, and to
demonstrate and encourage the use of potential intensity theory in tropical cyclone climatology analysis.

If you have any questions, comments, or feedback, please contact the developer or open an Issue in the repository. A paper detailing pyPI is published at Geoscientific Model Development.

Citation

pyPI was developed by Daniel Gilford and has been archived on Zenodo:

If you use pyPI in your work, please include the citations:

Gilford, D. M.: pyPI (v1.3): Tropical Cyclone Potential Intensity Calculations in Python, Geosci. Model Dev., 14, 2351–2369, https://doi.org/10.5194/gmd-14-2351-2021, 2021.

and

Gilford, D. M. 2020: pyPI: Potential Intensity Calculations in Python, pyPI v1.3. Zenodo. http://doi.org/10.5281/zenodo.3985975

Full pyPI Description

Please read pyPI_Users_Guide_v1.3.pdf for a full overview and details on pyPI. The description includes the pyPI background, a PI computation derivation, validation against the commonly-used MATLAB algorithm (pcmin), and a set of sample analyses.

Getting Started

pyPI requires Python version 3.7+ to run. It was originally written and tested with Python 3.7.6 and has been recently validated with Python 3.8.8 (as of 10 August 2022). To get pyPI up and running on your system, clone the repository and ensure that you have the required dependencies.

Installation

pyPI is packaged using the python package manager pip.

To install tcpypi from the command line:

pip install tcpypi

tcpyPI Dependencies

NumPy
Numba

Not required by tcpyPI---but highly recommended!---is the versatility in calculating PI over large datasets provided by xarray. Dependancy versions were originally handled by Dependabot, but the code was not resilient to these changes so they are currently defunct (as of 10 August 2022). Please notify me immediately if installation problems persist.

Python Implementation of "pc_min" (BE02 PI Calculator)

pi.py is the Python function which directly computes PI given atmospheric and ocean state variables (akin to the BE02 algorithm MATLAB implementation pc_min.m). Given input vector columns of environmental atmospheric temperatures (T) and mixing ratios (R) on a pressure grid (P), sea surface temperatures (SST), and mean sea-level pressures (MSL), the algorithm outputs potential intensity, the outflow level, the outflow temperature, and the minimum central pressure, and a flag that shows the status of the completed PI calculation. pyPI is an improvement on pcmin in that it handles missing values depending on user input flags.

Users who want to apply the PI calculation to a set of local environmental conditions need only to download pi.py, organize their data appropriately, and call the function to return outputs, e.g.:

(VMAX,PMIN,IFL,TO,LNB)=pi(SST,MSL,P,T,R)

Running a pyPI Sample

Included in the pyPI release is a sample script run_sample.py which runs global sample data from MERRA2 (in 2004) through pi.py, vectorizes the output, and performs several simple analyses. To run, simply:

python run_sample.py

and examine the outputs locally produced in full_sample_output.nc.

File Descriptions

Key files

pi.py - The primary function of pyPI, that computes and outputs PI (and associated variables) given atmospheric and ocean state variables.
run_sample.py - Example script that computes PI and accompanying analyses over the entire sample dataset

Data

sample_data.nc - Sample atmospheric and ocean state variable data and BE02 MATLAB output data; values are monthly averages over the globe from MERRA2 in 2004.
mdr.pk1 - Python pickled dictionary containing Main Development Region definitions from Gilford et al. (2017)
raw_sample_output.nc - Sample outputs from pi.py only created by run_sample.py
full_sample_output.nc - Full set of sample outputs from pi.py as well as sample analyses such as PI decomposition

Validation and Testing Notebooks

test_pi_calc.ipynb - Simple code showing a single call of pi.py and testing the speed of the algorithm
verify_pi.ipynb - Notebook validating/verifying pyPI outputs against BE02 MATLAB output data
sample_output_analyses.ipynb - Notebook showing examples of pyPI outputs and simple PI analyses

Misc.

utilities.py - Set of functions used in the pyPI codebase
constants.py - Set of meteorological constants used in the pyPI codebase
reference_calculations.m - Script used to generate sample BE02 MATLAB outout data from original MERRA2 files monthly mean; included for posterity and transperancy
pc_min.m - Original BE02 algorithm from MATLAB, adapted and used to produce analyses of Gilford et al. (2017; 2019)
clock_pypi.ipynb - Notebook estimating the time it takes to run pyPI on a laptop

Author

Daniel M. Gilford, PhD - Creation, Development, & Maintenance - GitHub

Contributor(s)

Daniel Rothenberg, PhD - Numba Optimization & Sample Code - GitHub

License

This project is licensed under the MIT License - see the LICENSE file for details

Acknowledgments

Kerry Emanuel (MIT) - Development of potential intensity theory; encouragement and permission to pursue Python implementation
Susan Solomon (MIT), Paul O'Gorman (MIT), Allison Wing (FSU) - Helpful conversations, advice, and suggestions on TC PI research
Dan Chavas (Purdue), Jonathan Lin (MIT), Raphael Rousseau-Rizzi (MIT) - Feedback on pyPI code and documentation

tcpypi's People

Contributors

Stargazers

Watchers

tcpypi's Issues

Pinned versions of xarray, numba, numpy

Hi @dgilford I just found out about this package via your post to the TC list! I notice that the versions of xarray, numba, and numpy are set to exactly 0.15.1, 0.48.0, and 1.18.1 respectively. I am just rolling out a new Python 3.8 environment for my department and would love to include this package, but this new environment uses 0.16.2, 0.52.0 and 1.19.5. Are there any test scripts/notebooks that could be run to see if tcpyPI will work ok with these updated libraries?

Thanks, Kevin

Python required version is not compatible with versions of packages in setup.py

Hi!

from setup.py, python_requires shows >=3.7

However, the xarray version 2022.3.0 and numpy 1.22 can only work on >=3.8.

python run_sample.py ERROR

Performing PI analyses...
Traceback (most recent call last):

File "/home/wamashudu/Videos/Potnetail intensity/tcpyPI-master/run_sample.py", line 148, in
ds2 = run_sample_analyses(ds,_mdrF,CKCD=0.9)
File "/home/wamashudu/Videos/Potnetail intensity/tcpyPI-master/run_sample.py", line 73, in run_sample_analyses
basins = pickle.load( open( _mdrF, "rb" ) )

AttributeError: Can't get attribute 'PandasIndexAdapter' on <module 'xarray.core.indexing' from '/home/wamashudu/anaconda3/lib/python3.9/site-packages/xarray/core/indexing.py'>

Issue in writing variables

@dgilford
I have successfully run the sample python file with my data

python run_sample.py**
**Beginning PI computations...
...PI computation complete and saved

Performing PI analyses...
...PI analyses complete and saved
PyPI sample run finished!**

After completing this I have tried to plot the VMAX variable in the out.nc but it showed the error like.
Warning: value for primary undef attribute "missing_value" not found
Cannot contour grid - all undefined values.

Can you please solve this.

Setting specific package versions may not be the best idea.

Currently setup.py shows:

    install_requires=[
        "xarray==2022.3.0",
        "numba==0.55.1",
        "numpy==1.22.3",
    ],

Unfortunately this will most certainly conflict with existing environments. For example #44. A more sensible option would be to specify compatible versions using ">=" instead of "==".

Rename pyPI to tcpyPI?

Nice package! I see that it's called tcpypi on pip, presumably because PyPI is already a package on pip. Given that the Python world is so familiar with PyPI as being a package repository, wouldn't it make sense to choose a different name for this project? tcpyPI seems not to be used for any other packages that I can see...

Error in tcpypi installation

Hi @dgilford
I started the installation of tcpypi it is successfully installed after that, I type
$python setup.py install
its shoi]wing

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tcpypi 1.3.4 requires numpy==1.19.5, but you have numpy 1.22.3 which is incompatible.

Give me the solution for this.
Thanks!

Problem when using pi.py

I'm trying to use pi() to get the VMAX. According to the comments the arguments P, TC, and R should all be one-dimensional arrays,
But utilities.t_ctok (TC) ,R * 0.001 makes TC, R look like int. I don't know how to use pi (). Can you give me an example data of P,TC,R?
Thanks a lot!

how to calculate outflow temperature level

Why could outflow temperature level be 0 hPa given that upper level pressure bound equals to 50 hPa?

Can we change file for "run_sample.py" from a pickle file to a more portable format?

When running run_sample.py on python 3.7 with:

netcdf4                   1.5.8                    pypi_0    pypi
numba                     0.55.1                   pypi_0    pypi
numpy                     1.21.6                   pypi_0    pypi
xarray                    0.20.2                   pypi_0    pypi

...PI computation complete and saved

Performing PI analyses...
Traceback (most recent call last):
  File "run_sample.py", line 148, in <module>
    ds2 = run_sample_analyses(ds,_mdrF,CKCD=0.9)
  File "run_sample.py", line 73, in run_sample_analyses
    basins = pickle.load( open( _mdrF, "rb" ) )
AttributeError: Can't get attribute 'PandasIndexAdapter' on <module 'xarray.core.indexing' from '/Users/arreyes/miniconda3/envs/mytcpypi/lib/python3.7/site-packages/xarray/core/indexing.py'>

The error occurs because the pickled file was saved with an unknown version of pickle and they are not supposed to be compatible (see e.g. pydata/xarray#5642) a better Idea could be to redistribute the file as netcdf or hdf5.

tcpypi won't install

I'm trying to install tcpypi, but when I type "pip install tcpypi", I get the following error:

  Building wheel for llvmlite (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/groups/oneillm/Laurel/miniconda3/envs/workenv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/setup.py'"'"'; __file__='"'"'/tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-fwoeenty
       cwd: /tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/
  Complete output (26 lines):
  running bdist_wheel
  /home/groups/oneillm/Laurel/miniconda3/envs/workenv/bin/python /tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/ffi/build.py
  LLVM version... Traceback (most recent call last):
    File "/tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/ffi/build.py", line 105, in main_posix
      out = subprocess.check_output([llvm_config, '--version'])
    File "/home/groups/oneillm/Laurel/miniconda3/envs/workenv/lib/python3.9/subprocess.py", line 424, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/home/groups/oneillm/Laurel/miniconda3/envs/workenv/lib/python3.9/subprocess.py", line 505, in run
      with Popen(*popenargs, **kwargs) as process:
    File "/home/groups/oneillm/Laurel/miniconda3/envs/workenv/lib/python3.9/subprocess.py", line 951, in __init__
      self._execute_child(args, executable, preexec_fn, close_fds,
    File "/home/groups/oneillm/Laurel/miniconda3/envs/workenv/lib/python3.9/subprocess.py", line 1821, in _execute_child
      raise child_exception_type(errno_num, err_msg, err_filename)
  FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config'
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/ffi/build.py", line 191, in <module>
      main()
    File "/tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/ffi/build.py", line 181, in main
      main_posix('linux', '.so')
    File "/tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/ffi/build.py", line 107, in main_posix
      raise RuntimeError("%s failed executing, please point LLVM_CONFIG "
  RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config
  error: command '/home/groups/oneillm/Laurel/miniconda3/envs/workenv/bin/python' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for llvmlite
  Running setup.py clean for llvmlite
Failed to build llvmlite
Installing collected packages: llvmlite, xarray, numba, tcpypi
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.38.0
    Uninstalling llvmlite-0.38.0:
      Successfully uninstalled llvmlite-0.38.0
    Running setup.py install for llvmlite ... error
    ERROR: Command errored out with exit status 1:
     command: /home/groups/oneillm/Laurel/miniconda3/envs/workenv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/setup.py'"'"'; __file__='"'"'/tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-fkaw76ne/install-record.txt --single-version-externally-managed --compile --install-headers /home/groups/oneillm/Laurel/miniconda3/envs/workenv/include/python3.9/llvmlite
         cwd: /tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/
    Complete output (29 lines):
    running install
    running build
    got version from file /tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/llvmlite/_version.py {'version': '0.34.0', 'full': 'c5889c9e98c6b19d5d85ebdd982d64a03931f8e2'}
    running build_ext
    /home/groups/oneillm/Laurel/miniconda3/envs/workenv/bin/python /tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/ffi/build.py
    LLVM version... Traceback (most recent call last):
      File "/tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/ffi/build.py", line 105, in main_posix
        out = subprocess.check_output([llvm_config, '--version'])
      File "/home/groups/oneillm/Laurel/miniconda3/envs/workenv/lib/python3.9/subprocess.py", line 424, in check_output
        return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
      File "/home/groups/oneillm/Laurel/miniconda3/envs/workenv/lib/python3.9/subprocess.py", line 505, in run
        with Popen(*popenargs, **kwargs) as process:
      File "/home/groups/oneillm/Laurel/miniconda3/envs/workenv/lib/python3.9/subprocess.py", line 951, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/home/groups/oneillm/Laurel/miniconda3/envs/workenv/lib/python3.9/subprocess.py", line 1821, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/ffi/build.py", line 191, in <module>
        main()
      File "/tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/ffi/build.py", line 181, in main
        main_posix('linux', '.so')
      File "/tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/ffi/build.py", line 107, in main_posix
        raise RuntimeError("%s failed executing, please point LLVM_CONFIG "
    RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config
    error: command '/home/groups/oneillm/Laurel/miniconda3/envs/workenv/bin/python' failed with exit code 1
    ----------------------------------------
  Rolling back uninstall of llvmlite
  Moving to /home/groups/oneillm/Laurel/miniconda3/envs/workenv/lib/python3.9/site-packages/llvmlite
   from /home/groups/oneillm/Laurel/miniconda3/envs/workenv/lib/python3.9/site-packages/~lvmlite
  Moving to /home/groups/oneillm/Laurel/miniconda3/envs/workenv/lib/python3.9/site-packages/llvmlite-0.38.0-py3.9.egg-info
   from /home/groups/oneillm/Laurel/miniconda3/envs/workenv/lib/python3.9/site-packages/~lvmlite-0.38.0-py3.9.egg-info
ERROR: Command errored out with exit status 1: /home/groups/oneillm/Laurel/miniconda3/envs/workenv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/setup.py'"'"'; __file__='"'"'/tmp/pip-install-1hum5keb/llvmlite_30d4375db3ee4b6d950f7cbfd970364a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-fkaw76ne/install-record.txt --single-version-externally-managed --compile --install-headers /home/groups/oneillm/Laurel/miniconda3/envs/workenv/include/python3.9/llvmlite Check the logs for full command output.

What might I do to fix this? Is there some other thing that I need to install? Or a module that I need?

Thanks!

Is xarray really a dependency?

Congrats on releasing this amazing package!

It looks like xarray is used for the examples, but your package itself (the contents of the pypi folder) doesn't ever import xarray.
https://github.com/dgilford/pyPI/search?q=xarray

So technically xarray is not a dependency.

If you were interested in building an xarray compatibility layer in pyPI, it would be pretty straightforward to wrap your functions using xarray.apply_ufunc. That way you could send xarray DataArrays directly in and out of pyPI.

We do something similar in our fastjmd95 package, which implements an ocean equation of state using numba: https://github.com/xgcm/fastjmd95/tree/master/fastjmd95

output variable values are all Zero

after successfully running the ..Sample PI with my data

.PI computation complete and saved
Performing PI analyses...
...PI analyses complete and saved
pyPI sample run finished!

all the output data show no value please how do i fix this

i attached my sample data
testpi.zip

data and validation

I'm using the sample_data.nc file you provide with the source code for validation but there are some small issues.

The time coordinates (month variable) are not specified in a standard way. For instance this variable does not have a calendar, and the units (Month Number) are not recognized by udunits.

Would it be possible to

change the units of the month variable to something more conventional such as "months since 1980-01-01"?
add a calendar attribute such as "standard" to the month variable?

In the run_sample.py you're passing q from the sample_data.nc file without conversion (here). The netcdf attributes in the file suggest that q is specific humidity while the tcpyPI documentation on states that a mixing ratio is required. Should there be a conversion applied?