mindsdb / mindsdb_evaluator Goto Github PK

View Code? Open in Web Editor NEW

9.0 9.0 4.0 6.06 MB

Model evaluation for Machine Learning pipelines.

License: GNU General Public License v3.0

Python 97.21% Makefile 2.79%

mindsdb_evaluator's People

Contributors

Stargazers

Watchers

Forkers

fairhopeweb minurapunchihewa mobasherah12 vijay-jaisankar

mindsdb_evaluator's Issues

Feature Request: Expected calibration error

I would like to raise a PR that implements the ECE metric in this file.

More specifically, the function proposed will take a list of softmax outputs, the y_true labels, and the number of bins. It will then calculate ECE as follows:

source

If this sounds like a good idea, please assign me to the same.

Thanks,
Vijay

Add evaluator class

This stateful mechanism should help enabling the following:

Track evaluations through time in a simple way
Store multiple-metric evals in a single object (users may want to eval a model with multiple metrics and consolidate all of these in a single artifact, which would be easier to produce from a single object)
Focus on reproducibility: there should be a simple mechanism that enables automatic logging of everything that is done to/with a marked instance
Scheduled support (i.e. cron jobs)? Not 100% sold on this one, it may be better to implement it within mdb proper

Incorrect label cast to integer

As reported by community. Using the Titanic dataset and one of {F1 score, recall, precision}. From MindsDB:

CREATE PREDICTOR mindsdb.titanic1
FROM files                      
(SELECT * FROM titanic)              
PREDICT Survived
USING
    accuracy_functions="['f1_score']";

Triggers the following error:

ValueError: pos_label=1 is not a valid label. It should be one of ['0', '1'], raised at: /usr/local/lib/python3.8/dist-packages/mindsdb/integrations/libs/ml_exec_base.py#135

Normal accuracy, balanced accuracy or roc auc score work okay.

Note: there is an additional issue with metrics that require parameter definition, like F1 score. Support for this is at the mindsdb_sql/mindsdb level (see respective issue for this).

Implement a bunch of accuracy funtions

Get AccStats to generate every single viable accuracy metrics it can think of for the problem types.

For starters let's do everything under sklearn's metrics + whatever @paxcema thinks is relevant for timeseries + @hakunanatasha if you have any ideas (or would like to warn about any sklearn metrics being bonk).

As with r2, I think it's better if we start with implementations that are wrappers over the sklearn function, and then add custom behavior if needed for various edge cases.

Also this shouldn't be equivalent to using all of those accuracy function for ensembling, the user should be able to specify a subset of functions that we ensemble on, but every single possible function should be computed by AccStats and added to the model data.

For now we can leave the defaults for ensembling to be the same.

Feature request: Top K accuracy score

Based on the Slack discussion, I would like to work on adding Top K accuracy to this project.

If you feel like this idea is worthwhile, please assign me to the same.

The corresponding PR will include

Exposing this metric from scikit-learn
Tests for the same

Feature Proposal: Test cases for `classification` and `regression`

I with to raise a PR that includes tests for the following methods:

evaluate_multilabel_accuracy
evaluate_regression_accuracy

As outlined in tests/test_forecast_accs.py, these will include at least

Random inputs
Test cases pertaining to small and large error values
Identical inputs and outputs (with zero error and hence maximum accuracy)

If this seems worthwhile, please assign me to the same.

Add support for Python 3.10

EDIT: achieved on last release.

[Bug] Mac M1 foes not have `np.float128`

If try to use mindsdb_evaluator at mac m1, then error will appear:

Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/alejandrovillegas/Documents/mindsdb/mindsdb/mindsdb/__main__.py", line 21, in <module>
    from mindsdb.api.http.start import start as start_http
  File "/Users/alejandrovillegas/Documents/mindsdb/mindsdb/mindsdb/api/http/start.py", line 9, in <module>
    from mindsdb.api.http.initialize import initialize_app
  File "/Users/alejandrovillegas/Documents/mindsdb/mindsdb/mindsdb/api/http/initialize.py", line 22, in <module>
    from mindsdb.api.http.namespaces.analysis import ns_conf as analysis_ns
  File "/Users/alejandrovillegas/Documents/mindsdb/mindsdb/mindsdb/api/http/namespaces/analysis.py", line 14, in <module>
    from mindsdb.api.mysql.mysql_proxy.classes.fake_mysql_proxy import FakeMysqlProxy
  File "/Users/alejandrovillegas/Documents/mindsdb/mindsdb/mindsdb/api/mysql/mysql_proxy/classes/fake_mysql_proxy/__init__.py", line 1, in <module>
    from .fake_mysql_proxy import FakeMysqlProxy
  File "/Users/alejandrovillegas/Documents/mindsdb/mindsdb/mindsdb/api/mysql/mysql_proxy/classes/fake_mysql_proxy/fake_mysql_proxy.py", line 3, in <module>
    from mindsdb.api.mysql.mysql_proxy.mysql_proxy import MysqlProxy
  File "/Users/alejandrovillegas/Documents/mindsdb/mindsdb/mindsdb/api/mysql/mysql_proxy/mysql_proxy.py", line 57, in <module>
    from mindsdb.api.mysql.mysql_proxy.executor import Executor
  File "/Users/alejandrovillegas/Documents/mindsdb/mindsdb/mindsdb/api/mysql/mysql_proxy/executor/__init__.py", line 1, in <module>
    from .mysql_executor import Executor
  File "/Users/alejandrovillegas/Documents/mindsdb/mindsdb/mindsdb/api/mysql/mysql_proxy/executor/mysql_executor.py", line 6, in <module>
    from mindsdb.api.executor.command_executor import ExecuteCommands
  File "/Users/alejandrovillegas/Documents/mindsdb/mindsdb/mindsdb/api/executor/command_executor.py", line 8, in <module>
    from mindsdb_evaluator.accuracy.general import evaluate_accuracy
  File "/Users/alejandrovillegas/Library/Python/3.9/lib/python/site-packages/mindsdb_evaluator/__init__.py", line 1, in <module>
    from mindsdb_evaluator.accuracy import *  # noqa
  File "/Users/alejandrovillegas/Library/Python/3.9/lib/python/site-packages/mindsdb_evaluator/accuracy/__init__.py", line 6, in <module>
    from mindsdb_evaluator.accuracy.general import evaluate_accuracy, evaluate_accuracies
  File "/Users/alejandrovillegas/Library/Python/3.9/lib/python/site-packages/mindsdb_evaluator/accuracy/general.py", line 16, in <module>
    SCORE_TYPES = (float, np.float16, np.float32, np.float64, np.float128,
  File "/Users/alejandrovillegas/Library/Python/3.9/lib/python/site-packages/numpy/__init__.py", line 311, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'float128'

mindsdb / mindsdb_evaluator Goto Github PK

mindsdb_evaluator's People

Contributors

Stargazers

Watchers

Forkers

mindsdb_evaluator's Issues

Feature Request: Expected calibration error

Add evaluator class

Incorrect label cast to integer

Implement a bunch of accuracy funtions

Feature request: Top K accuracy score

Feature Proposal: Test cases for `classification` and `regression`

Add support for Python 3.10

[Bug] Mac M1 foes not have `np.float128`

Improve custom accuracy function support

Move AccStats and ConfStats in

Potential error on RMSE

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent