Comments (1)
ivan-khvostishkov
commented on May 23, 2024
Hi, @airlangga-gunawan-faculty , thanks for reporting the issue and apologies for the delayed response due to my PTO.
You are right that IMDS access is blocked, but SSM Agent should never reach this stage.
Basically, the successful initialization inside SageMaker Studio looks like this:
sm-ssh-ide: Starting SSM agent
--
Initializing new seelog logger
New Seelog Logger Creation Complete
2023/10/12 11:13:48 Found config file at /etc/amazon/ssm/amazon-ssm-agent.json.
Applying config override from /etc/amazon/ssm/amazon-ssm-agent.json.
2023/10/12 11:13:48 processing appconfig overrides
2023/10/12 11:13:48 Found config file at /etc/amazon/ssm/amazon-ssm-agent.json.
Applying config override from /etc/amazon/ssm/amazon-ssm-agent.json.
2023/10/12 11:13:48 processing appconfig overrides
2023-10-12 11:13:48 INFO Proxy environment variables:
2023-10-12 11:13:48 INFO https_proxy:
2023-10-12 11:13:48 INFO http_proxy:
2023-10-12 11:13:48 INFO no_proxy:
2023-10-12 11:13:48 INFO Checking if agent identity type OnPrem can be assumed
2023-10-12 11:13:48 INFO Agent will take identity from OnPrem
2023-10-12 11:13:48 INFO [amazon-ssm-agent] using named pipe channel for IPC
2023-10-12 11:13:48 INFO [amazon-ssm-agent] using named pipe channel for IPC
2023-10-12 11:13:48 INFO [amazon-ssm-agent] using named pipe channel for IPC
2023-10-12 11:13:48 INFO [amazon-ssm-agent] amazon-ssm-agent - v3.2.1705.0
2023-10-12 11:13:48 INFO [amazon-ssm-agent] OS: linux, Arch: amd64
2023-10-12 11:13:48 INFO [amazon-ssm-agent] Starting Core Agent
2023-10-12 11:13:48 INFO [CredentialRefresher] credentialRefresher has started
2023-10-12 11:13:48 INFO [CredentialRefresher] Starting credentials refresher loop
2023-10-12 11:13:48 INFO [CredentialRefresher] Credentials ready
2023-10-12 11:13:48 INFO [CredentialRefresher] Next credential rotation will be in 29.99631267835 minutes
2023-10-12 11:13:49 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] [WorkerProvider] Worker ssm-agent-worker is not running, starting worker process
2023-10-12 11:13:49 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] [WorkerProvider] Worker ssm-agent-worker (pid:14716) started
2023-10-12 11:13:49 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] Monitor long running worker health every 60 seconds
As you can see, it should be taking the OnPrem identity, not the EC2. This behaviour is verified by unit tests that are reproducible, e.g. take a look at the IDE test for version 2.1.0.
So you should be looking for the difference between the test environment and your environment. Few things to check inside SageMaker Studio Kernel, e.g. from the image terminal:
1/ That /etc/amazon/ssm/amazon-ssm-agent.json file exists and corresponds to sm-setup-ssh .
2/ The AWS_CONTAINER_CREDENTIALS_RELATIVE_URI variable exists inside kernel gateway app and you don't accidentally reset it inside your lifecycle configuration script.
Check the LifecycleConfigOnStart log in CloudWatch, it should contain the following lines
sm-save-env: Dumping environment
...
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI='/_sagemaker-instance-credentials/abcdef0123456789example'
This env var belongs to Container credential provider (https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html), in contrast to the IMDS provider.
3/ Try the iPython notebook instead of the lifecycle config to check if it makes any difference.
If the above steps won't help, please, create a technical support case and refer them to this issue, so that I can take a look closer together with the support team.
from sagemaker-ssh-helper.
Related Issues (20)
- [Issue] When use the MMS host model. e.g. HuggingFace Model, no any info in cloudwatch log and can not use ssh HOT 10
- Issue trying with a SageMaker Notebook: "sagemaker-ssh-helper:SSMManager:SSH Helper not yet started? Retrying." HOT 6
- Is there any additional cost&limits related to using (SSM) Session Manager? HOT 4
- Error on `dpkg` when running `sm-local-configure` HOT 4
- How to enable cloudwatch logs for SSM HOT 2
- [Feature] Support Hugging Face Accelerate for training HOT 1
- Issue: Invalid bucket name "sagemaker.config INFO - Fetched defaults config from location: ": Bucket name must match the regex
- Is there any plan to support it in BJS/ZHY? HOT 1
- [Issue]failed to find agent identity HOT 6
- [Issue] `sm-local-configure` breaks on MacOS HOT 4
- Issue] STS client is not using regional endpoints HOT 2
- [Feature] Make switching instances in SageMaker Studio and PyCharm more smooth
- [Feature] Support HF accelerate and DeepSpeed for inference HOT 1
- Thoughts on using a configuration management framework? HOT 6
- sm-local-configure only works with bash like installations - no Powershell/CMD support / Windows support at all HOT 4
- Are scripts supposed to work on SageMaker notebook instances? HOT 12
- How to install VSCode, other apps in WebVNC view? HOT 2
- JupyterServer URL suffix when tunnelling into KernelGateway app HOT 2
- Notebook `SageMaker_SSH_Notebook.ipynb` fails due to docker-compose HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sagemaker-ssh-helper.