Logging experiments to S3 bucket about clearml-server HOT 4 CLOSED

Hi @qraleq , as you mentioned, one obvious solution is to map /opt/trains/data/fileserver to your s3 bucket with s3fs.

The other maybe easier is simply to set the files_server to your s3 bucket.

Edit /trains.conf and replace this line
files_server: "http://localhost:8081"
With
files_server: "s3://mybucket/logs/"

And do not forget to add the AWS S3 credentials :)

BTW:
The console logs and the graphs themselves (not the debug images) are stored into the elastic search database in the trains-server

from clearml-server.

@bmartinn Thank you for your answer! I'm interested in saving the logs and graphs to an S3 bucket, so I can access it from different machines. For example, I would run one instance for training, and I would like to have access to the experiments even when the training instance is offline using another instance which is available 24/7.

What's the best way to achieve this?

from clearml-server.

Yes @qraleq that is exactly why we designed the trains-server to be deployed as a server (as opposed to running it per user).

If your trains-server instance is running on the cloud, then you can do much more that view the logs, you can also share them, you can even remotely stop an experiment from the web UI (Abort), and not to forget the trains-agent addition, for creating your own cluster.
Anyway :)

Where are you running the current trains-server? You can quickly set up an instance on amazon with the trains-server AMI or with a simple docker-compose on any GCP / Azure cpu instance.

Once you have that, just configure the ~/trains.conf to point to the cloud trains-server, and you should be good to go.

What do you think?

from clearml-server.

@bmartinn Thank you very much for your answer, I understand how you've structured things now!

from clearml-server.