atrcheema / ai4water Goto Github PK
View Code? Open in Web Editor NEWframework for developing machine (and deep) learning models for structured data
Home Page: https://ai4water.readthedocs.io
License: MIT License
framework for developing machine (and deep) learning models for structured data
Home Page: https://ai4water.readthedocs.io
License: MIT License
Hi,
First, thank you for this package. I am planning to use it in a research project over the coming year at least, and look forward to do so, and hopefully offer to contribute back.
I tried to set up a virtual environment (venv or conda) suitable for ai4water, and it was a bit bumpy. Not unexpected mind you, having been there several times. I also tried installing both tensorflow and pytorch in the same environment, which was perhaps overly ambitious and risky.
A virtual environment with a pip install tensorflow
is easy to set up, but this does assume that the right versions of libcdnn and cuda are installed at the OS level, which is not a trivial endeavour.
I've had success in the past reliably installing a conda environment for pytorch and fastai, including suitable versions cuDNN libraries.
I am planning to document ways to reliably install environments with a mix of docker, conda, pip (as necessary, the lesser the better) to get ai4water running,
If you have any advice on what environments you have to run ai4water, I'd welcome tips.
J-M
When using the ML model from ai4water, if the model has two output variables and the training completes successfully, there might be an issue during the prediction step when calling model.predict(data=dataset) from the trained model. This error occurs because the model is facing a problem during the inverse transformation of the output variables.
The error is; ValueError: Shape of passed values is (3110, 1), indices imply (3110, 2)
This kind of problem can be solved by replacing the Model class of ai4water by MyModel class as below
class MyModel(Model):
"""
A model with correct inverse transfomration of y
"""
def _inverse_transform_y(self, true_outputs, predicted):
"""inverse transformation of y/labels for both true and predicted"""
if self.config['y_transformation']:
from ai4water.preprocessing import Transformations
y_transformer = Transformations.from_config(self.config['y_transformer_'])
if self.config['y_transformation']: # only if we apply transformation on y
# both x,and true_y were given
true_outputs = self.__inverse_transform_y(true_outputs, y_transformer)
# because observed y may have -ves or zeros which would have been
# removed during fit and are put back into it during inverse transform, so
# in such case predicted y should not be treated by zero indices or negative indices
# of true y. In other words, parameters of true y should not have impact on inverse
# transformation of predicted y.
predicted = self.__inverse_transform_y(predicted, y_transformer, postprocess=False)
return true_outputs, predicted
def __inverse_transform_y(self,
y,
transformer,
method="inverse_transform",
postprocess=True
)->np.ndarray:
"""inverse transforms y where y is supposed to be true or predicted output
from model."""
# todo, if train_y had zeros or negatives then this will be wrong
if isinstance(y, np.ndarray):
# it is ndarray, either num_outs>1 or quantiles>1 or forecast_len>1 some combination of them
if y.size > len(y):
if y.ndim == 2:
# todo, following line was problematic
y = getattr(transformer, method)(y, postprocess=postprocess)
else:
# (exs, outs, quantiles) or (exs, outs, forecast_len) or (exs, forecast_len, quantiles)
for out in range(y.shape[1]):
for q in range(y.shape[2]):
y[:, out, q] = getattr(transformer, method)(y[:, out, q],
postprocess=postprocess).reshape(-1, )
else: # 1d array
y = getattr(transformer, method)(y, postprocess=postprocess)
# y can be None for example when we call model.predict(x=x),
# in this case we don't know what is y
elif y is not None:
raise ValueError(f"can't inverse transform y of type {type(y)}")
return y
For the record; I will submit a PR for these kind of things, if appropriate.
Running compare_ml.ipynb at commit a8dae50, comparisons.fit(run_type="dry_run")
leads to:
"prior should be 'uniform' or 'log-uniform' got log"
visibly in skopt
, using version 0.9.0
This is likely related to the change made for scikit-optimize #1017
the folder's content is not like what you described, in ./data/ ,can not find "data/all_data_30min.csv"
also in ./models/ can not find some files which names like Model, ConvLstm etc.
I cannot import "from models import Model" like this.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.