Issue with Sky Spot Controller when launched from GCP VM about skypilot HOT 7 OPEN

1. We do not explicitly mount ~/.sky to VM/container
2. SSH key changes on new VM/container.

The error does not happen when I run the same container from my local (launches a different spot controller each time), but when launched from a VM (with a service account), it tries to access the same spot controller and fails.

Thanks for sharing more details @subhamde8247! One hypothesis of this would be that both the linux username and the python -c "import socket; print(socket.gethostname())" is the same for the multiple containers, causing the user hash we generated to identify different machines based on the two values being the same, which leads to using the same spot controller.

To confirm the hypothesis, it would be nice to check if cat ~/.sky/user_hash has the same value across multiple VM/containers.

There are several workarounds:

1. share the SSH key across multiple VM/container by explicitly upload/mounting those keys to them.
2. Or, randomly generate a user hash for each VM/container whenever it is firstly provisioned, by randomly generating the ~/.sky/user_hash: python -c "import uuid; print(uuid.uuid4().hex[:8])" > ~/.sky/user_hash

We will also look into the issue and see if the username and the python -c "import socket; print(socket.gethostname())" not sufficient for identifying a user : )

from skypilot.

One hack we have been using now: use gcloud compute instances list to list all VMs that match the pattern sky-spot-controller- at the end of each run and deleting them.

Wondering if I am missing something or if there is a better solution.

from skypilot.

from a docker container inside GCP VM that has a service account attached to it

Thanks for the report @subhamde8247! Could you share some details of this client VM? For example,

• what does “gcloud auth list” show inside this container? Does it only have the service account, or also some static credential files?
• On each day, is the spot launch triggered from a different container on this client VM, or the same container?

from skypilot.

1. “gcloud auth list” only shows the service account
2. The client VM is deleted each day after running, and a new VM is created next day, and a new container started within that VM. So spot launch is triggered from a different container each day attached to a new VM instance.

from skypilot.

@subhamde8247 Got it. Some followups:

• Do you mount the same ~/.sky to the new VM/container everyday? This would explain why the new VM/container reuse the same spot controller.
• Does the SSH key, ~/.ssh/sky-key{.pub}, change on the new VM/container? A newly generated key would explain why the connection to the same spot controller VM is unsuccessful.
  • If this is the case, a fix should be mounting the same ~/.ssh/sky-key{.pub} to the new VM/container everyday. This way the same spot controller can be reused, and during idle periods it'd be autostopped to save costs.

from skypilot.

1. We do not explicitly mount ~/.sky to VM/container
2. SSH key changes on new VM/container.

The error does not happen when I run the same container from my local (launches a different spot controller each time), but when launched from a VM (with a service account), it tries to access the same spot controller and fails.

from skypilot.

Confirmed that cat ~/.sky/user_hash is same for multiple runs of docker container when launched from multiple VMs. However, the hashes are different for multiple runs when same container is run from my local.

explicitly upload/mounting those keys

yeah, this will add some complexity of storing these keys in GCP secret manager, and properly loading them during the container start-up each day.

randomly generate a user hash for each VM/container

we don't mind having a new spot controller for each daily job. The only issue - old spot controllers are not auto-downed and we are left with a bunch of stopped spot controller instances in our VM list (which we have to manually delete). If available, a --down option for spot controller would work for us.

from skypilot.