VSCode disconnects after credentials refresh. about sagemaker-ssh-helper HOT 6 OPEN

Hi, @harish-kamath , very nice that you like this library! I have few questions:
1/ When you say InternalServerError: We encountered an internal error. Please try again - which log file exactly do you see this message at? Is it in CloudWatch?
2/ Do you know which process generates this message, e.g., is there any prefix in front of this line?
3/ What SageMaker component you are connecting to, e.g., SageMaker Training or Studio, or Inference?

from sagemaker-ssh-helper.

@ivan-khvostishkov

Upon further digging - I'm not sure if it is actually the credentials (only).

I noticed that the ssm agent will first pull AWS credentials from the environment variables - so I tried including explicit AWS access key and secret key in my training job. The logs still show that the credentials are being refreshed, but the machine doesn't actually crash ~1m after the credentials are refreshed anymore. However, now it just crashes otherwise (even if there is nothing running, so no chance that it's a resource issue).

1. This error message is on the Sagemaker console, under Training Jobs. Here's an example:
   

And here is the last cloudwatch log:

(Note that in this case, I did not do sm-wait stop, but it doesn't actually matter for this error. It occurs even if I do that)

2. There's no prefix or process unfortunately. Since it just crashes the machine, and there's no persistent storage, I'm not sure how I can actually debug after a crash either.

3. Sagemaker Training Jobs

from sagemaker-ssh-helper.

On the bright side, it no longer crashes always after 30 mins of being connected. However, it is still crashing within an hour.

from sagemaker-ssh-helper.

Never mind, just got another crash in <30 minutes.

I'm pretty sure it is still this package, because connecting over plain SSH is still fine and never causes a crash.

from sagemaker-ssh-helper.