Unable to deploy huggingface-llm 1.3.3 about sagemaker-python-sdk HOT 12 CLOSED

We will release 1.3.3 through DLC with an ETA of tomorrow.

The associated SDK change can be tracked through #4335. However, you can also just reference the image URIs specifically to avoid waiting for an SDK release. Available tags and sample image URI can be found here: https://github.com/aws/deep-learning-containers/releases?q=tgi&expanded=true.

from sagemaker-python-sdk.

1.3.1 should be released in the sdk by now. #4314

@LvffY can you share why 1.3.3? This could help us accelerate the release

from sagemaker-python-sdk.

I believe HuggingFace has requested that we hold until they are able to merge the fixes in for huggingface/text-generation-inference#1334.

from sagemaker-python-sdk.

hi! A fix has been applied for huggingface/text-generation-inference#1334. cc @philschmid

from sagemaker-python-sdk.

I believe @philschmid mentions that we're waiting for this PR to be accepted: #4314

from sagemaker-python-sdk.

@cfregly I don't think we're waiting for the same version because @philschmid seemed to wait for 1.3.1 version while I'D like to see the 1.3.3 version

But I may look into this PR to see if I can update the sdk myself if anyone answer here :)

from sagemaker-python-sdk.

@philschmid The main idea is to be able to run Mistral 0.2 models. For now, with all supported versions are throwing the issue described in the huggingface repostory.

Looking at the comments, we see that this should be fixed with this PR which is included in the latest released version 1.3.3

from sagemaker-python-sdk.

Confirmed that 1.3.1 (SageMaker Python SDK 2.200.1) still throws the same error.

from sagemaker-python-sdk.

Noting that the reason we are not able to fetch a v1.3.3 image through the SDK is because there is no actual DLC release in itself for v1.3.3. It is not a bug in the SDK from what I have read so far.

from sagemaker-python-sdk.

Noting that the reason we are not able to fetch a v1.3.3 image through the SDK is because there is no actual DLC release in itself for v1.3.3. It is not a bug in the SDK from what I have read so far.

So what should be the way to go here ?

from sagemaker-python-sdk.

@amzn-choeric Is it possible to release 1.3.4 through DLC as well?

1.3.4 has a fix for this issue which will allow some Mistral models (with flash attention v2) to run on instances with non Ampere GPU

from sagemaker-python-sdk.

I believe that would need to be included in a new release version as deemed appropriate where we can then discuss with HuggingFace about the next steps after.

With regards to the issue at hand, though, it does look like the SDK change has been merged. Thus, closing the issue.

from sagemaker-python-sdk.