Comments (4)
Michaelvll
commented on September 23, 2024
Could you share the detailed logs?
from skypilot.
cblmemo
commented on September 23, 2024
Could you share the detailed logs?
log for test_gcp_images:
(basically, the job is always pending and timeout in the end)
+ sky launch -y -c t-gcp-images-c9 --image-id skypilot:gpu-debian-10 --cloud gcp tests/test_yamls/minimal.yaml
D 03-19 23:45:15 skypilot_config.py:136] Using config path: /home/txia/.sky/config.yaml
D 03-19 23:45:15 skypilot_config.py:140] Config loaded:
D 03-19 23:45:15 skypilot_config.py:140] {'gcp': {'vpc_name': 'skypilot-vpc'}}
D 03-19 23:45:15 skypilot_config.py:150] Config syntax check passed.
Task from YAML spec: tests/test_yamls/minimal.yaml
D 03-19 23:45:17 optimizer.py:262] #### Task<name=min>(run='conda env list\n')
D 03-19 23:45:17 optimizer.py:262] resources: GCP(image_id=skypilot:gpu-debian-10) ####
D 03-19 23:45:18 optimizer.py:284] Defaulting the task's estimated time to 1 hour.
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-central1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-central1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-central1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-central1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-east1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-east1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-east1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-east5': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-east5': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-east5': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-west1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-west1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-west1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'me-west1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'me-west1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'me-west1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west4': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west4': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west4': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'northamerica-northeast1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'northamerica-northeast1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'northamerica-northeast1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'northamerica-northeast2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'northamerica-northeast2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'northamerica-northeast2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-north1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-north1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-north1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-west4': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-west4': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-west4': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-east4': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-east4': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-east4': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-east1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-east1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-east1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.4
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west8': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west8': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west8': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west9': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west9': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west9': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-southwest1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-southwest1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-southwest1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-south1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-south1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-south1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-south1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-south1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-south1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-south2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-south2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-south2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-west2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-west2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-west2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-west3': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-west3': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'us-west3': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'me-central1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'me-central1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'me-central1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-southeast1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-southeast1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-southeast1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-northeast1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-northeast1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-northeast1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-northeast2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-northeast2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-northeast2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-northeast3': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-northeast3': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-northeast3': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-central2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:18 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:18 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:18 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-central2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-central2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west3': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west3': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west3': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west12': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west12': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west12': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'africa-south1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'africa-south1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'africa-south1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-southeast2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-southeast2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-southeast2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west6': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west6': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west6': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-east2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-east2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'asia-east2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.5
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'australia-southeast1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'australia-southeast1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'australia-southeast1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'australia-southeast2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'australia-southeast2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'australia-southeast2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'southamerica-west1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'southamerica-west1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'southamerica-west1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west10': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west10': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'europe-west10': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'southamerica-east1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'southamerica-east1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'southamerica-east1': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'me-central2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'me-central2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
D 03-19 23:45:19 optimizer.py:305] resources: GCP(n2-standard-8, image_id={'me-central2': 'skypilot:gpu-debian-10'})
D 03-19 23:45:19 optimizer.py:324] estimated_runtime: 3600 s (1.0 hr)
D 03-19 23:45:19 optimizer.py:328] estimated_cost (not incl. egress): $0.6
I 03-19 23:45:19 optimizer.py:691] == Optimizer ==
I 03-19 23:45:19 optimizer.py:702] �[1mTarget:�[0m minimizing cost
I 03-19 23:45:19 optimizer.py:714] �[1mEstimated cost: �[0m$0.4 / hour
I 03-19 23:45:19 optimizer.py:714]
I 03-19 23:45:19 optimizer.py:837] �[1mConsidered resources (1 node):�[0m
I 03-19 23:45:19 optimizer.py:907] ----------------------------------------------------------------------------------------------
I 03-19 23:45:19 optimizer.py:907] CLOUD INSTANCE vCPUs Mem(GB) ACCELERATORS REGION/ZONE COST ($) CHOSEN
I 03-19 23:45:19 optimizer.py:907] ----------------------------------------------------------------------------------------------
I 03-19 23:45:19 optimizer.py:907] GCP n2-standard-8 8 32 - us-central1-a 0.39 �[32m ✔�[0m
I 03-19 23:45:19 optimizer.py:907] ----------------------------------------------------------------------------------------------
I 03-19 23:45:19 optimizer.py:907]
Running task on cluster t-gcp-images-c9...
D 03-19 23:45:19 cloud_vm_ray_backend.py:4176] cluster_ever_up: False
D 03-19 23:45:19 cloud_vm_ray_backend.py:4177] record: None
I 03-19 23:45:19 cloud_vm_ray_backend.py:4237] �[36mCreating a new cluster: 't-gcp-images-c9' [1x GCP(n2-standard-8, image_id={'us-central1': 'skypilot:gpu-debian-10'})].�[0m
I 03-19 23:45:19 cloud_vm_ray_backend.py:4237] Tip: to reuse an existing cluster, specify --cluster (-c). Run `sky status` to see existing clusters.
I 03-19 23:45:22 cloud_vm_ray_backend.py:1364] To view detailed progress: �[1mtail -n100 -f /home/txia/sky_logs/sky-2024-03-19-23-45-17-725890/provision.log�[0m
D 03-19 23:45:23 backend_utils.py:841] Using ssh_proxy_command: None
D 03-19 23:45:23 skypilot_config.py:93] User config: gcp.vpc_name -> skypilot-vpc
D 03-19 23:45:26 provisioner.py:168]
D 03-19 23:45:26 provisioner.py:168]
D 03-19 23:45:26 provisioner.py:168] ==================== Provisioning ====================
D 03-19 23:45:26 provisioner.py:168]
D 03-19 23:45:26 provisioner.py:169] Provision config:
D 03-19 23:45:26 provisioner.py:169] {
D 03-19 23:45:26 provisioner.py:169] "provider_config": {
D 03-19 23:45:26 provisioner.py:169] "type": "external",
D 03-19 23:45:26 provisioner.py:169] "module": "sky.skylet.providers.gcp.GCPNodeProvider",
D 03-19 23:45:26 provisioner.py:169] "region": "us-central1",
D 03-19 23:45:26 provisioner.py:169] "availability_zone": "us-central1-a",
D 03-19 23:45:26 provisioner.py:169] "cache_stopped_nodes": true,
D 03-19 23:45:26 provisioner.py:169] "project_id": "skypilot-375900",
D 03-19 23:45:26 provisioner.py:169] "vpc_name": "skypilot-vpc",
D 03-19 23:45:26 provisioner.py:169] "use_internal_ips": false,
D 03-19 23:45:26 provisioner.py:169] "disable_launch_config_check": true
D 03-19 23:45:26 provisioner.py:169] },
D 03-19 23:45:26 provisioner.py:169] "authentication_config": {
D 03-19 23:45:26 provisioner.py:169] "ssh_user": "gcpuser",
D 03-19 23:45:26 provisioner.py:169] "ssh_private_key": "~/.ssh/sky-key"
D 03-19 23:45:26 provisioner.py:169] },
D 03-19 23:45:26 provisioner.py:169] "docker_config": {},
D 03-19 23:45:26 provisioner.py:169] "node_config": {
D 03-19 23:45:26 provisioner.py:169] "labels": {
D 03-19 23:45:26 provisioner.py:169] "skypilot-user": "txia"
D 03-19 23:45:26 provisioner.py:169] },
D 03-19 23:45:26 provisioner.py:169] "machineType": "n2-standard-8",
D 03-19 23:45:26 provisioner.py:169] "disks": [
D 03-19 23:45:26 provisioner.py:169] {
D 03-19 23:45:26 provisioner.py:169] "boot": true,
D 03-19 23:45:26 provisioner.py:169] "autoDelete": true,
D 03-19 23:45:26 provisioner.py:169] "type": "PERSISTENT",
D 03-19 23:45:26 provisioner.py:169] "initializeParams": {
D 03-19 23:45:26 provisioner.py:169] "diskSizeGb": 256,
D 03-19 23:45:26 provisioner.py:169] "sourceImage": "projects/deeplearning-platform-release/global/images/common-cu113-v20221215",
D 03-19 23:45:26 provisioner.py:169] "diskType": "zones/us-central1-a/diskTypes/pd-balanced"
D 03-19 23:45:26 provisioner.py:169] }
D 03-19 23:45:26 provisioner.py:169] }
D 03-19 23:45:26 provisioner.py:169] ],
D 03-19 23:45:26 provisioner.py:169] "metadata": {
D 03-19 23:45:26 provisioner.py:169] "items": [
D 03-19 23:45:26 provisioner.py:169] {
D 03-19 23:45:26 provisioner.py:169] "key": "ssh-keys",
D 03-19 23:45:26 provisioner.py:169] "value": "gcpuser:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCmsJ+roPZhYum6AN8FBYGfDDo+B0oGIjuqGfMHjx3ehzDOlb8tKXbSP0WTZG72Zv9+R7smvXcEOQHZXvALndRzfjh1Thfe68SG9HBMYVliQtYBI9NHyw4PKBVgMDxmdCinsZCPPJtZpAn++Rzwux5URcI3zbGvTwr8SlnmEFhqIOdMrCtoyLSPI/WIV3M41lf/syMYqCjTw1oIogzW5rKTW/16Iuw0tsg9IeEjdAy0yq6SVe19El5yTxxFrrmL1atxTJlPn/pEb0HW1UmIl3Q7+Bt8ebWyrd6RIP/nzm2EjeDAgIGnjwAImlzcJytoem5kKbE+zkF+MmrBAbAM0A4H"
D 03-19 23:45:26 provisioner.py:169] }
D 03-19 23:45:26 provisioner.py:169] ]
D 03-19 23:45:26 provisioner.py:169] }
D 03-19 23:45:26 provisioner.py:169] },
D 03-19 23:45:26 provisioner.py:169] "count": 1,
D 03-19 23:45:26 provisioner.py:169] "tags": {},
D 03-19 23:45:26 provisioner.py:169] "resume_stopped_nodes": true
D 03-19 23:45:26 provisioner.py:169] }
I 03-19 23:45:26 provisioner.py:76] �[1mLaunching on GCP us-central1�[0m (us-central1-a)
D 03-19 23:45:32 skypilot_config.py:93] User config: gcp.vpc_name -> skypilot-vpc
D 03-19 23:45:59 provisioner.py:104]
D 03-19 23:45:59 provisioner.py:104] Waiting for instances of 't-gcp-images-c9' to be ready...
D 03-19 23:46:00 provisioner.py:123] Instances of 't-gcp-images-c9' are ready after 0 retries.
D 03-19 23:46:00 provisioner.py:127]
D 03-19 23:46:00 provisioner.py:127] Provisioning 't-gcp-images-c9' took 34.67 seconds.
D 03-19 23:46:01 provisioner.py:577]
D 03-19 23:46:01 provisioner.py:577]
D 03-19 23:46:01 provisioner.py:577] ==================== System Setup After Provision ====================
D 03-19 23:46:01 provisioner.py:577]
D 03-19 23:46:06 provisioner.py:423] Provision record:
D 03-19 23:46:06 provisioner.py:423] {
D 03-19 23:46:06 provisioner.py:423] "provider_name": "gcp",
D 03-19 23:46:06 provisioner.py:423] "region": "us-central1",
D 03-19 23:46:06 provisioner.py:423] "zone": "us-central1-a",
D 03-19 23:46:06 provisioner.py:423] "cluster_name": "t-gcp-images-c9-4a07",
D 03-19 23:46:06 provisioner.py:423] "head_instance_id": "t-gcp-images-c9-4a07-head-71wnpjfa-compute",
D 03-19 23:46:06 provisioner.py:423] "resumed_instance_ids": [],
D 03-19 23:46:06 provisioner.py:423] "created_instance_ids": [
D 03-19 23:46:06 provisioner.py:423] "t-gcp-images-c9-4a07-head-71wnpjfa-compute"
D 03-19 23:46:06 provisioner.py:423] ]
D 03-19 23:46:06 provisioner.py:423] }
D 03-19 23:46:06 provisioner.py:423] Cluster info:
D 03-19 23:46:06 provisioner.py:423] {
D 03-19 23:46:06 provisioner.py:423] "instances": {
D 03-19 23:46:06 provisioner.py:423] "t-gcp-images-c9-4a07-head-71wnpjfa-compute": [
D 03-19 23:46:06 provisioner.py:423] {
D 03-19 23:46:06 provisioner.py:423] "instance_id": "t-gcp-images-c9-4a07-head-71wnpjfa-compute",
D 03-19 23:46:06 provisioner.py:423] "internal_ip": "10.128.0.58",
D 03-19 23:46:06 provisioner.py:423] "external_ip": "34.42.214.207",
D 03-19 23:46:06 provisioner.py:423] "tags": {
D 03-19 23:46:06 provisioner.py:423] "skypilot-user": "txia",
D 03-19 23:46:06 provisioner.py:423] "ray-cluster-name": "t-gcp-images-c9-4a07",
D 03-19 23:46:06 provisioner.py:423] "skypilot-cluster-name": "t-gcp-images-c9-4a07",
D 03-19 23:46:06 provisioner.py:423] "skypilot-head-node": "1",
D 03-19 23:46:06 provisioner.py:423] "ray-node-type": "head"
D 03-19 23:46:06 provisioner.py:423] },
D 03-19 23:46:06 provisioner.py:423] "ssh_port": 22
D 03-19 23:46:06 provisioner.py:423] }
D 03-19 23:46:06 provisioner.py:423] ]
D 03-19 23:46:06 provisioner.py:423] },
D 03-19 23:46:06 provisioner.py:423] "head_instance_id": "t-gcp-images-c9-4a07-head-71wnpjfa-compute",
D 03-19 23:46:06 provisioner.py:423] "docker_user": null,
D 03-19 23:46:06 provisioner.py:423] "ssh_user": null,
D 03-19 23:46:06 provisioner.py:423] "custom_ray_options": null
D 03-19 23:46:06 provisioner.py:423] }
D 03-19 23:46:07 provisioner.py:446]
D 03-19 23:46:07 provisioner.py:446] Waiting for SSH to be available for 't-gcp-images-c9' ...
D 03-19 23:46:07 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:07 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:08 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:08 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:09 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:09 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:10 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:10 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:11 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:11 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:13 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:13 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:14 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:14 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:15 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:15 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:16 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:16 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:17 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:17 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:18 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:18 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:19 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:19 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:21 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:21 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:22 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:22 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:23 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:23 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:24 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:24 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:25 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:25 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:26 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:26 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:28 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:28 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:29 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:29 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:30 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:30 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:31 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:31 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:32 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:32 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:33 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:33 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:35 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:35 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:36 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:36 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:37 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:37 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:38 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:38 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:39 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:39 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:40 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:40 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:41 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:41 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:43 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:43 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:44 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:44 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:45 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:45 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:46 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:46 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:47 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:47 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:48 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:48 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:50 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:50 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:51 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:51 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:52 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:52 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:53 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:53 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:54 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:54 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:55 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:55 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:57 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:57 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:58 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:58 provisioner.py:400] Retrying in 1 second...
D 03-19 23:46:59 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:46:59 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:00 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:00 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:01 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:01 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:02 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:02 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:03 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:03 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:05 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:05 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:06 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:06 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:07 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:07 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:08 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:08 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:09 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:09 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:10 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:10 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:12 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:12 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:13 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:13 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:14 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:14 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:15 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:15 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:16 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:16 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:17 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:17 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:19 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:19 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:20 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:20 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:21 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:21 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:22 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:22 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:23 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:23 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:24 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:24 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:26 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:26 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:27 provisioner.py:328] Waiting for SSH to 34.42.214.207. Try: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime. Error: �[1mConnectionRefusedError:�[0m [Errno 111] Connection refused
D 03-19 23:47:27 provisioner.py:400] Retrying in 1 second...
D 03-19 23:47:28 provisioner.py:349] Waiting for SSH using command: ssh -T -i '~/.ssh/sky-key' [email protected] -p 22 -o StrictHostKeyChecking=no -o ConnectTimeout=10s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 uptime
D 03-19 23:47:30 provisioner.py:449] SSH Conection ready for 't-gcp-images-c9'
I 03-19 23:47:30 provisioner.py:451] �[32mSuccessfully provisioned or found existing instance.�[0m
D 03-19 23:47:54 provisioner.py:521] Starting Ray on the entire cluster.
I 03-19 23:48:09 provisioner.py:553] �[32mSuccessfully provisioned cluster: t-gcp-images-c9�[0m
D 03-19 23:48:09 cloud_vm_ray_backend.py:2267] Using provided external IPs: ['34.42.214.207']
D 03-19 23:48:09 cloud_vm_ray_backend.py:2284] Cached external IPs do not match with the newly fetched ones: cached (None), new (['34.42.214.207'])
D 03-19 23:48:09 cloud_vm_ray_backend.py:2296] Using provided internal IPs: ['10.128.0.58']
I 03-19 23:48:10 cloud_vm_ray_backend.py:3076] �[36mRunning setup on 1 node.�[0m
running setup
I 03-19 23:48:17 cloud_vm_ray_backend.py:3089] �[32mSetup completed.�[0m
D 03-19 23:48:17 cloud_vm_ray_backend.py:3091] Setup took 7.489387512207031 seconds.
D 03-19 23:48:20 cloud_vm_ray_backend.py:546] Added Task with options: num_cpus=0.5, scheduling_strategy=ray.util.scheduling_strategies.PlacementGroupSchedulingStrategy(placement_group=pg, placement_group_bundle_index=0)
I 03-19 23:48:27 cloud_vm_ray_backend.py:3171] Job submitted with Job ID: �[1m1�[0m
I 03-20 06:48:30 log_lib.py:406] �[33mStart streaming logs for job 1.�[0m
INFO: Job finished (status: FAILED).
�[0mShared connection to 34.42.214.207 closed.
I 03-19 23:53:12 cloud_vm_ray_backend.py:3205] �[36mJob ID: �[1m1�[0m
I 03-19 23:53:12 cloud_vm_ray_backend.py:3205] To cancel the job: �[1msky cancel t-gcp-images-c9 1�[0m
I 03-19 23:53:12 cloud_vm_ray_backend.py:3205] To stream job logs: �[1msky logs t-gcp-images-c9 1�[0m
I 03-19 23:53:12 cloud_vm_ray_backend.py:3205] To view the job queue: �[1msky queue t-gcp-images-c9�[0m
I 03-19 23:53:12 cloud_vm_ray_backend.py:3301]
I 03-19 23:53:12 cloud_vm_ray_backend.py:3301] �[36mCluster name: �[1mt-gcp-images-c9�[0m
I 03-19 23:53:12 cloud_vm_ray_backend.py:3301] To log into the head VM: �[1mssh t-gcp-images-c9�[0m
I 03-19 23:53:12 cloud_vm_ray_backend.py:3301] To submit a job: �[1msky exec t-gcp-images-c9 yaml_file�[0m
I 03-19 23:53:12 cloud_vm_ray_backend.py:3301] To stop the cluster: �[1msky stop t-gcp-images-c9�[0m
I 03-19 23:53:12 cloud_vm_ray_backend.py:3301] To teardown the cluster: �[1msky down t-gcp-images-c9�[0m
D 03-19 23:53:12 skypilot_config.py:136] Using config path: /home/txia/.sky/config.yaml
D 03-19 23:53:12 skypilot_config.py:140] Config loaded:
D 03-19 23:53:12 skypilot_config.py:140] {'gcp': {'vpc_name': 'skypilot-vpc'}}
D 03-19 23:53:12 skypilot_config.py:150] Config syntax check passed.
Clusters
NAME LAUNCHED RESOURCES STATUS AUTOSTOP COMMAND
t-gcp-image-id-dict-zone-f1 11 secs ago 1x GCP(n2-standard-8, image_id={'us-central1': 'skypilot:cpu-debian-10... UP - sky launch -y -c t-gcp-im...
t-gcp-images-c9 5 mins ago 1x GCP(n2-standard-8, image_id={'us-central1': 'skypilot:gpu-debian-10... UP - sky launch -y -c t-gcp-im...
t-gcp-image-id-dict-40 6 mins ago 1x GCP(n2-standard-8, image_id={'us-central1': 'skypilot:cpu-debian-10... UP - sky exec t-gcp-image-id-d...
sky-spot-controller-4a0782e9 5 mins ago 1x AWS(m6i.2xlarge, disk_size=50) UP 10m sky spot launch -n t-spot...
sky-serve-controller-4a0782e9 21 hrs ago 1x GCP(n2-standard-4, disk_size=200, ports=['30001-30100']) STOPPED 10m sky serve up @temp/temp.y...
* 1 cluster has auto{stop,down} scheduled. Refresh statuses with: sky status --refresh
�[?25h+ sky logs t-gcp-images-c9 1 --status
D 03-19 23:53:15 skypilot_config.py:136] Using config path: /home/txia/.sky/config.yaml
D 03-19 23:53:15 skypilot_config.py:140] Config loaded:
D 03-19 23:53:15 skypilot_config.py:140] {'gcp': {'vpc_name': 'skypilot-vpc'}}
D 03-19 23:53:15 skypilot_config.py:150] Config syntax check passed.
�[33mGetting job status...�[0m
Job 1: FAILED
D 03-19 23:53:20 skypilot_config.py:136] Using config path: /home/txia/.sky/config.yaml
D 03-19 23:53:20 skypilot_config.py:140] Config loaded:
D 03-19 23:53:20 skypilot_config.py:140] {'gcp': {'vpc_name': 'skypilot-vpc'}}
D 03-19 23:53:20 skypilot_config.py:150] Config syntax check passed.
D 03-19 23:53:22 cloud_vm_ray_backend.py:3679] Provisioner version: ProvisionerVersion.SKYPILOT using new provisioner for teardown.
D 03-19 23:53:26 instance.py:40] handlers: [<class 'sky.provision.gcp.instance_utils.GCPComputeInstance'>]
D 03-19 23:53:28 instance.py:51] handler_to_instances: defaultdict(<class 'list'>, {<class 'sky.provision.gcp.instance_utils.GCPComputeInstance'>: ['t-gcp-images-c9-4a07-head-71wnpjfa-compute']})
D 03-19 23:53:28 instance.py:528] Terminating instance: t-gcp-images-c9-4a07-head-71wnpjfa-compute.
D 03-19 23:53:30 instance.py:124] wait_for_compute_zone_operation: Waiting for operation operation-1710917609886-614120c6094b1-97deaf28-25476495 to finish...
D 03-19 23:53:31 instance_utils.py:411] Waiting GCP operation operation-1710917609886-614120c6094b1-97deaf28-25476495 to be ready ...
D 03-19 23:53:46 metadata_utils.py:115] Remove metadata of cluster t-gcp-images-c9-4a07.
D 03-19 23:53:46 metadata_utils.py:115] Remove metadata of cluster t-gcp-images-c9.
Terminating cluster t-gcp-images-c9...done.
Terminating 1 cluster ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
�[31mFailed�[0m.
Reason: sky logs t-gcp-images-c9 1 --status
Log: less /tmp/gcp_images-wts9d_04.log
from skypilot.
Michaelvll
commented on September 23, 2024
Ahh, this is a known issue fixed by skypilot-org/skypilot-catalog#62. Could you remove ~/.sky/catalogs/v5/gcp and try again?
from skypilot.
cblmemo
commented on September 23, 2024
Ahh, this is a known issue fixed by skypilot-org/skypilot-catalog#62. Could you remove
~/.sky/catalogs/v5/gcpand try again?
This fix the issue. Thanks!
Besides, why does the catalog is not automatically updated..?
from skypilot.
Related Issues (20)
- [Core] Job cancellation sometimes returns 255
- [TPU] Support reservations for TPU
- Spot instance support for runpod.
- [AWS] S3 doesn't work on non-AWS controllers when using AWS SSO
- How to setup test for new cloud?
- [GCP] ultra disk fail to work with L4 GPUs HOT 1
- [Docker] Docker image as runtime fails on GCP HOT 4
- [Serve] Inline update for readiness_probe HOT 1
- [GCP][Disk] `sky launch --cpus 2 --cloud gcp --disk-tier ultra` not showing real reason HOT 1
- [R2] Mounting a R2 storage fail to work HOT 1
- [R2] R2 not supported as source in file_mounts HOT 1
- [UX] File sync ignores files in `.gitignore` causing debugging on remote fails
- [Storage][Serve] Surface the error message when SkyServe failed on workdir storage HOT 2
- [Feature] Autostop definition in YAML? HOT 3
- [UX][Serve] Combine the uploaded files for Sky Serve
- [Serve] Endless attempts to find instances
- [Docs] SkyServe integration with AWS CloudWatch
- [Docs][Serve/Spot] Adding doc for running controller on local k8s cluster
- [UX] Make `labels` work in task.yaml without the requirement for specifying `cloud`
- [UX/AWS] When SSO expired, `sky status` shows a lot SSO error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from skypilot.