Unable to use custom Keras metrics with TFX evaluator component about model-analysis HOT 4 CLOSED

In general, this sounds like an issue where the custom module is not available on the Beam worker, even though it is available in your main program (where the Beam pipeline is constructed). My first thought is to suggest you try setting the module_file parameter in the Evaluator component with an absolute path to your module file. This will package the module_file and distribute it to the Beam workers.

A couple clarifications which might help narrow things down:

• How are you invoking the Evaluator component? Are you running it in a TFX pipeline? Ideally we could find a portable way to reproduce this issue.
• Was this custom metric setup previously working, and it broke with TFMA version 0.22.2? Or has this never worked?
• Does your real metric module name (not what I assume is the placehorlder, my.module_file.path.metrics) include the literal, tf.keras.metrics? If so, then it might have been affected by the change you linked, though this seems pretty unlikely.

from model-analysis.

Have some updates here -

• We publish the custom metrics code in an extra_package that is sent to beam workers, and can confirm this package has the code, and that from logs, the workers install that extra_package.
• The setup was not previously working, it was a first attempt at implementing custom metrics
• It does not include the literal tf.keras.metrics which is why I thought it wouldnt be affected

That said, we have managed to update the code to use TFX 0.27 (TFMA 0.27) and magically the code seems to work 🤷 (no other changes to the code related to the evaluator component, custom metrics code or packaging thereof) I have no idea what change might have caused the fix but perhaps we can close the issue.

from model-analysis.

Well, glad it's fixed. It would be nice to understand the root cause, but it's probably not a high priority unless this issue reappears for others. Thanks for taking the time to report the issue and follow up.

from model-analysis.

Closing this based on above comment trace,Thanks.

from model-analysis.