drsteve / pyforecasttools Goto Github PK

Feature Request
The dictionary returned by plot.rocCurve has Figure, Axes, POD, POFD, and Thresholds.
It'd be extremely useful to add the area under the curve (AUC) as a measure of model performance.

Currently the implementation uses Contingency2x2.fromBoolean() and loops over thresholds creating a new Contingency2x2 object each time. It might be good to consider directly supporting probabilistic predictions, possibly in a similar manner to fromBoolean - this could them allow updating of the contingency table by updating the threshold as the probabilities would be stored with the object (and would allow bootstrapping CIs, same as fromBoolean).
AUC could then be calculated without the ROC plotting.

The reliability diagram (in the plot module) currently automatically determines the number of bins to use for the reliabilty diagram by the Freedman-Diaconis method, with a hard upper limit on the number of bins (for graphical purposes).

For datasets with limited numbers of events (i.e., imbalanced datasets) this can lead to bin sizes being much smaller than desired. It would be useful to allow specification of the number of bins, or bin selection method, in the call to verify.plot.reliabilityDiagram.

Current documentation is sparse. It would be helpful to users to provide:

• Simple example calculations in the docstrings
• Some example use cases, possibly as scripts in an examples folder, or as Jupyter notebooks, or as pages in documentation (at the github.io web page)

Installation produces 5 files: dependency_links.txt, PKG-INFO, requires.txt, SOURCES.txt, and top_level.txt. While I'm not sure why all of these files are needed (especially top_level.txt), I think they should be added to .gitignore.

Feature Request
Currently the plot.reliabilityDiagram feature plots the predicted probability against empirical probability, as well as adding a histogram of the predicted probabilities.

It'd be extremely useful have options to add:

• a horizontal line for the climatological probability of event occurrence
• a shaded region of positive Brier skill score

The documentation is ugly. The docstring formatting isn't presently being rendered correctly by Sphinx.

The workflow for the github.io webpage for this project currently follows https://daler.github.io/sphinxdoc-test/includeme.html

ETA: Documentation is at https://drsteve.github.io/PyForecastTools, and auto-updates whenever the gh-pages branch of this repo is updated.

Missing unit tests for the functions.

Steve,

minor warning that is likely not important but thought I should put it here.

numpy=1.18.3

verify.metrics.medSymAccuracy(fitted_model(x), data)

/Users/balarsen/miniconda3/envs/python3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:746: UserWarning: Warning: 'partition' will ignore the 'mask' of the MaskedArray.
  a.partition(kth, axis=axis, kind=kind, order=order)

After installing from pip, some of the functions such as bias, etc, are not attached to verify when importing. After uninstalling from pip and reinstalling from github, all functions are importing correctly. I suspect that switching to absolute import statements in init would fix this on the pip side.

from verify.metrics import *
from verify.categorical import *
from verify. import plot