Track train/validation metrics in the Sagemaker dashboard when doing remote tuning about syne-tune HOT 11 CLOSED

OK, this means you use remote tuning, and so you'll need what I wrote as a task into our backlog.
Maybe you could open an issue and give some more details of what metrics you'd like to be reported? I can then work on it, this is a good feature to have.

from syne-tune.

As for metrics output during training, this is the responsibility of the training code. We provide examples for SM backend.

from syne-tune.

One interesting thing we do not have, would be to emit metrics to the dashboard for remote tuning. That is a good point

from syne-tune.

Can you link the example you mention above?
The only mention of metrics I found in the FAQs is this: https://syne-tune.readthedocs.io/en/latest/faq.html#what-are-the-metrics-reported-by-default-when-calling-the-reporter

from syne-tune.

https://github.com/awslabs/syne-tune/blob/main/examples/launch_height_sagemaker.py#L84

from syne-tune.

It is the metric_definitions field for a SM estimator.
Of course, you also need to report these metrics. But we do that with the Reporter anyway.

from syne-tune.

I put the idea to output metrics for HPO experiment for remote tuning in our backlog, because I like it. I'll probably do this soon. This will then work if you use remote tuning with local backend, for example.

from syne-tune.

I am not sure I completely follow your comments but we have a syne-tune Tuner running in a SM training job and using a machine with multiple GPUs (one per worker).

Anyway, adding a question to the FAQs will probably help others to find the solution quickly.

from syne-tune.

I updated the issue title instead of opening a new one.

We need metrics which are tied to the metric optimized with HPO (e.g., best so far, value in the last evaluation), and a convention for customers to define their own (e.g., all the metrics provided to the syne-tune reporter). On top of that I think it would be nice to have metrics which are tied to HPO optimizers internals and can be optimizer-specific (e.g., num configurations completed, num configurations pruned).

from syne-tune.

PR #590 will close this issue

from syne-tune.

@610v4nn1 The PR is offering most of what you asked for. We have a backlog item to configure a CloudWatch setup so these metrics are displayed nicely. Right now, the docs offer some guidance.

from syne-tune.