Unclear EvalConfig nomenclature for model outputs about model-analysis HOT 4 CLOSED

It is a bit confusing. In the case of Estimators, a single output model would output multiple prediction keys (probabilities, classes, etc) but only one of the keys was intended to be used as the prediction (e.g. 'probabilities', 'logistic', etc). If someone customized the name of that key then they would need to update their EvalConfig.model_spec to set the 'prediction_key'. The estimator could also be multi-head (multi-output) in which case there would be multiple dicts each with a set of keys (the 'head' name would correspond to the EvalConfig.metrics_spec.output_name' and the EvalConfig.model_spec.prediction_key` would refer to the entry representing the prediction under each head).

For keras there is only a single key per output so only the 'EvalConfig.metrics_spec.output_name' applies for these models.

Long story short, AFAICT you have a multi-output model with two possible outputs 'mu' and 'sigma', so you should set the 'EvalConfig.metrics_spec.output_name' to either (or both) outputs. However, if your setup is such that the prediction is always either 'mu' and 'sigma' and the other key is just extra information, then you should set the 'EvalConfig.model_spec.prediction_key' instead.

from model-analysis.

Ah, I see. That is rather confusing indeed 😂 What worked for me was the following:

eval_config = tfma.EvalConfig(
    model_specs=[
        tfma.ModelSpec(
            label_key="my_label_key",
            prediction_key="mu" # setting the prediction key
        )
    ],
    metrics_specs=[
        tfma.MetricsSpec(
            output_names=["mu"], # and the output names
            metrics=[tfma.MetricConfig(
                class_name="MeanSquaredError",
                config='"dtype": "float32", "name": "mse"'
            )]
        )
    ],
    slicing_specs=[tfma.SlicingSpec()]
)
eval_shared_model = tfma.default_eval_shared_model(saved_model_dir, eval_config=eval_config)

What happens if I want to use a custom metric with a update_state method that has a nonstandard function signature? The typical function signature is:

def update_state(self, y_true, y_pred, sample_weight=None):

However, my custom metric uses the following signature:

def update_state(self, y_true, y_pred, some, other, arguments, sample_weight=None):

I can't seem to figure out (from the TFMA documentation) how to specify a custom metric with a update_state that has been overridden. As such, when I run the model analysis, I encounter the following error:

TypeError: in user code:
    TypeError: tf__update_state() missing 3 required positional arguments: 'some', 'other', and 'arguments'

Do we only have the option to pass kwargs to initialise the metric via EvalConfig.MetricConfig.config?

from model-analysis.

For keras metrics we currently only support metrics that can be passed to model.compile (i.e. that take the standard arguments: y_true, y_pred, sample_weight). In the future we will support other metrics (i.e. those added via model.add_metric), but those will need to be added as part of your model otherwise we don't know what arguments to pass.

from model-analysis.

I understand. Guess I'll close this issue for now since it doesn't seem like we'll be able to get to a resolution. Thanks @mdreves

from model-analysis.