How to extend the container with custom inference code? about sagemaker-pytorch-inference-toolkit HOT 9 CLOSED

I seem to have figured out the issue here. It seems that when the image is being spun up as an endpoint, there is some code that removes all files inside /opt/ml/model BEFORE copying and unzipping model.tar.gz from S3. If you package your custom inference code in model.tar.gz then it will work. I have spent many hours figuring this out. Also, your saved model should be called model_0.pth as that is what MMS will pass to your custom inference code as the parameter model_dir.

model.tar.gz
|- model_0.pth
|- code
| |- inference.py

This sort of behavior from the code is in fact really concerning because it means that in order to change your custom inference code, you need to repackage your model data. This really makes working with this container setup in a CICD way clunky and hacky at best.

At the very least, we should be adding what I mentioned above to the README or documentation for this project since its incredibly unclear how you would add your own custom inference code.

from sagemaker-pytorch-inference-toolkit.

Expanding on @josephjohansson's answer, to extend pre-built images with custom inference code for TensorFlow serving, you need to package your inference code so that the model.tar.gz extracts like this:

To pack it that way, you should run
tar -czf model.tar.gz <model_dir> <code_dir>

I found this out by using SageMaker's TensorFlowModel API to package a model with an entry_point and inspecting its contents.

They should really expand details here so an unsuspecting dev would know what's happening in the background.

from sagemaker-pytorch-inference-toolkit.

Documentation on using this package with custom inference code, including model artifact structure, as well as how SageMaker runs the container, can be found here:

• https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html#model-directory-structure
• https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html

I'll summarize certain relevant details that should clear up confusion in this thread.

When deploying your model to a SageMaker Endpoint, you first create a SageMaker Model from your model artifact, either using SageMaker's CreateModel API (https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html) or with the SageMaker Python SDK (https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html#create-a-pytorchmodel-object). You then create a SageMaker endpoint. When SageMaker deploys the model, SageMaker downloads and unpacks the model artifact and mounts the unpacked directory at /opt/ml/model in the container as a readonly Docker volume.

from sagemaker-pytorch-inference-toolkit.

Anyone please?

from sagemaker-pytorch-inference-toolkit.

Yes, the example you are using was for a different container.

Since we switched to use MMS as our model server inside the container the code now needs to be copied under /opt/ml/model/code/ .

from sagemaker-pytorch-inference-toolkit.

Hi,
I was facing the same issue, below is the Dockerfile I am using. It seems the model_fn is still not being read. @artmatsak Were you able to solve this issue?
@nadiaya Any help here would be appreciated!

ARG REGION=us-east-1

FROM 763104351884.dkr.ecr.$REGION.amazonaws.com/pytorch-inference-eia:1.3.1-cpu-py36-ubuntu16.04

ENV PATH="/opt/ml/code:${PATH}"

COPY /src /opt/ml/model/code

RUN pip3 install -r /opt/ml/code/requirements.txt

ENV SAGEMAKER_SUBMIT_DIRECTORY /opt/ml/model/code

ENV SAGEMAKER_PROGRAM serve.py

from sagemaker-pytorch-inference-toolkit.

hello. Just out of the curiosity : in current release, do we have the possibility to have multi/model capabilities for pytorch containers?

from sagemaker-pytorch-inference-toolkit.

hello. Just out of the curiosity : in current release, do we have the possibility to have multi/model capabilities for pytorch containers?

@velenciucsergiu Multi-model support is available for PyTorch 1.4.0 and 1.5.0:

from sagemaker-pytorch-inference-toolkit.

Hi @josephjohansson
Does your solution also relate to deploy Inference model within Docker containers? and if so how?

from sagemaker-pytorch-inference-toolkit.