UncertaintyPlayground

Installation

Requirements:

Python >= 3.8
PyTorch == 2.0.1
GPyTorch == 1.10
Numpy == 1.24.0
Seaborn == 0.12.2

Use PyPI to install the package:

pip install uncertaintyplayground

or alterntively, to use the development version, install directly from GitHub:

pip install git+https://github.com/unco3892/UncertaintyPlayground.git

Usage

You can train and visualize the results of the models in the following way (this example uses the California Housing dataset from Sklearn):

from uncertaintyplayground.trainers.svgp_trainer import SparseGPTrainer
from uncertaintyplayground.trainers.mdn_trainer import MDNTrainer
from uncertaintyplayground.predplot.svgp_predplot import compare_distributions_svgpr
from uncertaintyplayground.predplot.mdn_predplot import compare_distributions_mdn
from uncertaintyplayground.predplot.grid_predplot import plot_results_grid
import torch
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split

# Load the California Housing dataset
california = fetch_california_housing()

# Convert X and y to numpy arrays of float32
X = np.array(california.data, dtype=np.float32)
y = np.array(california.target, dtype=np.float32)

# Set random seed for reproducibility
np.random.seed(42)
torch.manual_seed(1)

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# SVGPR: Initialize and train a SVGPR model with 100 inducing points
california_trainer_svgp = SparseGPTrainer(X_train, y_train, num_inducing_points=100, num_epochs=30, batch_size=512, lr=0.1, patience=3)
california_trainer_svgp.train()

# MDN: Initialize and train an MDN model
california_trainer_mdn = MDNTrainer(X_train, y_train, num_epochs=100, lr=0.001, dense1_units=50, n_gaussians=10)
california_trainer_mdn.train()

# SVPGR: Visualize the SVGPR's predictions for multiple instances
plot_results_grid(trainer=california_trainer_svgp, compare_func=compare_distributions_svgpr, X_test=X_test, Y_test=y_test, indices=[900, 500], ncols=2)

# MDN: Visualize the MDN's predictions for multiple instances
plot_results_grid(trainer=california_trainer_mdn, compare_func=compare_distributions_mdn, X_test=X_test, Y_test=y_test, indices=[900, 500], ncols=2)

You can find another example for MDN in the examples folder.

Contributors

This library is maintained by Ilia Azizi (University of Lausanne). Any other contributors are welcome to join! Feel free to get in touch with (contact links on my website).

Citation

If you use this package in your research, please cite our work:

UncertaintyPlayground: A Fast and Simplified Python Library for Uncertainty Estimation , Ilia Azizi, arXiv:2310.15281

@misc{azizi2023uncertaintyplayground,
      title={UncertaintyPlayground: A Fast and Simplified Python Library for Uncertainty Estimation}, 
      author={Ilia Azizi},
      year={2023},
      eprint={2310.15281},
      archivePrefix={arXiv},
      primaryClass={stat.ML}
}

License

Please see the project MIT licensed here.

[Feature Request] List of possible improvements

⚡ Feature Request

Motivation

Adding a list of general features to be implemented for the future versions.

Pitch

Major ideas

Flexible Model Architectures: For the MDN model, it would be beneficial to make the neural network architecture modular, allowing for the incorporation of different layers and activation functions. For the SVGP model, adding the option to use different kernel functions could extend the model's flexibility.
Improved Noise Modelling: Introducing the capability to use different types of noise in the MDN model could significantly improve the quality of uncertainty estimates.
Classification Capabilities: Both the MDN and SVGP models could be extended to support binary and multi-class classification. This would involve modifying the likelihood function and the performance metric. This would, however, require extensive theoretical and empirical validation as MDNs and SVGPRs are not traditionally used for classification tasks.
Hardware Utilization: The package could benefit from implementing multi-GPU support, which would allow for more efficient training of large models on large datasets. Optimizing the parallel data loading process for maximized CPU utilization could significantly improve overall performance.
Benchmarking Performance: In the further iterations of this package, the performance, both in terms of speed and accuracy of prediction, can be measured against other models. For instance, one can compare our approach with a traditional GPR for larger and smaller datasets since, as already discussed, GPR has an algorithmic complexity of (O(N^3)) and does not scale well beyond a few hundred observations. Additionally, conformal prediction, and MAPIE can be compared with our techniques.

Minor ideas

Remove any irrelevant dependencies that are already bundled via the main package + Make sure that the library is compatible with the current version or newer versions of the dependencies : Allow for better integrating with improved dependency management.
Improved Code Documentation: The addition of type hints to the docstrings would offer better clarity and type checking, enhancing the readability and maintainability of the codebase. Additionally, the documentation can benefit from more examples.
Add support (documentation) for R programming language through reticulate: Many users are interested also in having such techniques in R, which is particularly relevant since there are not many packages for uncertainty prediction.

unco3892 / uncertaintyplayground Goto Github PK