Received server error (500) from primary and could not load the entire response body from endpoint about realtime-fraud-detection-with-gnn-on-dgl HOT 4 OPEN

@zhjwy9343 is the data scentist for authoring those Notebook. James, could you have a look at those problems?

from realtime-fraud-detection-with-gnn-on-dgl.

@zinebtabet Could you add the detailed reproduciable steps how you using the code?

from realtime-fraud-detection-with-gnn-on-dgl.

I used the same code as you have in the SageMaker repository. The only thing I modified was the Docker file since I am in EU West 3. I set it up like this: ARG IMAGE_REPO=763104351884.dkr.ecr.eu-west-3.amazonaws.com FROM $IMAGE_REPO/pytorch-training:1.11.0-cpu-py38-ubuntu20.04-sagemaker ENV PATH="/opt/ml/model:${PATH}" ENV SAGEMAKER_SUBMIT_DIRECTORY /opt/ml/model COPY * /opt/ml/code/ ENV SAGEMAKER_PROGRAM fd_sl_train_entry_point.py RUN pip install dgl dglgo -f https://data.dgl.ai/wheels/repo.html

I used the same version I specified in the Docker file for the deployment as well. Then I invoked my endpoint with the Lambda function. Once I executed the test event, I received this error. I have had this error for over a month now. I will provide screenshots of the error:

the error in the lambda test event: Traceback (most recent call last): File "/var/task/lambda_function.py", line 442, in lambda_handler pred_prob = invoke_endpoint_with_idx(endpointname = ENDPOINT_NAME, target_id = transaction_id, subgraph_dict = subgraph_dict, n_feats = transaction_embed_value_dict) File "/var/task/lambda_function.py", line 314, in invoke_endpoint_with_idx response = runtime.invoke_endpoint(EndpointName=endpointname, File "/opt/python/lib/python3.8/site-packages/botocore/client.py", line 508, in _api_call return self._make_api_call(operation_name, kwargs) File "/opt/python/lib/python3.8/site-packages/botocore/client.py", line 911, in _make_api_call raise error_class(parsed_response, operation_name)

from realtime-fraud-detection-with-gnn-on-dgl.

Hello @zxkane,

Thank you for the nice project. I learn a lot from it.

We use the SageMaker notebook & studio to reproduce the project. The model was created and repackaged successfully and the endpoints of them work well. Suddenly, (in a middle of a demo) the endpoint didn't respond.

Right now, we have only one model that has a workable endpoint which is trained and repackaged with SageMaker notebook.

From then none of our endpoints (the models created with notebook or studio) does not work anymore.

I appreciate any help or suggestion!

from realtime-fraud-detection-with-gnn-on-dgl.