[Feature] Copying file to an instance from local about sagemaker-ssh-helper HOT 10 CLOSED

Hi, @ssmall41, thank you for asking. Indeed, this is something that was already requested by some other users, and I already recorded the TODO item for it:

Let’s keep this issue open for tracking your request until it’s implemented in a next release.

from sagemaker-ssh-helper.

Sure. The procedure should look like this (run from your local terminal in two separate tabs):

1. sm-local-ssh-training connect <training-job-name>
2. rsync -rvh -e 'ssh -i ~/.ssh/sagemaker-ssh-gw -p 11022' --delete project_dir/source_dir/ root@localhost:/opt/ml/code/

from sagemaker-ssh-helper.

Thanks @ivan-khvostishkov. The TODO mentions SageMaker Studio. Would it also be possible to do this with Training Jobs or Endpoints?

from sagemaker-ssh-helper.

Yes, of course, thank you for checking. It is exactly the request for training jobs and endpoints. SageMaker Studio in this context is an example of IDE that doesn't have a native capability to copy files to remote machine. PyCharm, for example, can synchronize files automatically with SFTP, and this additional API brings little value to it. What IDE do you use on your local host?

from sagemaker-ssh-helper.

Thank you! I am using PyCharm. If I understand correctly, PyCharm's file sync should work with the ssh helper already. Is it possible to transfer files through the aws cli?

from sagemaker-ssh-helper.

Yes, for PyCharm easier to use the native IDE deployment capability. For AWS CLI, I don’t know an easy way to do it. But if you use CLI helper scripts like sm-local-ssh-training, you’re be able today to use rsync CLI command to copy files to remote, as a workaround.

from sagemaker-ssh-helper.

Thanks @ivan-khvostishkov Can you provide an example of how one would use rsync with the helper sm-local-ssh-training?

from sagemaker-ssh-helper.

Thanks Ivan for the suggestion and the code. One minor issue with the rsync command above is that the files transferred to the remote instance end up owned by root, not ec2-user. This permission change prevents any further modification from the notebook end (although nothing stops us from making changes locally and synchronize them with rsync). Fortunately, this can be easily fixed by passing --chown 1000:1000 to rsync:

rsync -rvh -e 'ssh -i ~/.ssh/sagemaker-ssh-gw -p 11022' --delete \
  --chown 1000:1000 \
  project_dir/source_dir/ root@localhost:/opt/ml/code

(1000 seems to be the UID of ec2-user, but I am not sure how robust this is).

from sagemaker-ssh-helper.

Hi, @djmarti , thank you for the note. As I understand, you're trying to edit the files on the SageMaker notebook instance (i.e. using SageMaker_SSH_Notebook.ipynb). In this case, the owner of the file should be the 1000, and your command will work pretty well, indeed. I will think how to make it work in PyCharm, too.

from sagemaker-ssh-helper.

Hi Ivan. Apologies for the delay in responding. You are right, I was able to copy files from my local machine to the notebook instance, but I was unable to modify the newly copied files in the notebook instance. The reason was the owner of the newly copied files was no longer ec2-user.

from sagemaker-ssh-helper.