Received server error (500) from primary and could not load the entire response body from endpoint

[zinebtabet]

[ERROR] ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from primary and could not load the entire response body. See https://eu-west-3.console.aws.amazon.com/cloudwatch/home?region=eu-west-3#logEventViewer:group=/aws/sagemaker/Endpoints/pytorch-inference-2023-07-10-09-51-02-299 in account 086892845792 for more information. Traceback (most recent call last): File "/var/task/lambda_function.py", line 442, in lambda_handler pred_prob = invoke_endpoint_with_idx(endpointname = ENDPOINT_NAME, target_id = transaction_id, subgraph_dict = subgraph_dict, n_feats = transaction_embed_value_dict) File "/var/task/lambda_function.py", line 314, in invoke_endpoint_with_idx response = runtime.invoke_endpoint(EndpointName=endpointname, File "/opt/python/lib/python3.8/site-packages/botocore/client.py", line 508, in _api_call return self._make_api_call(operation_name, kwargs) File "/opt/python/lib/python3.8/site-packages/botocore/client.py", line 911, in _make_api_call raise error_class(parsed_response, operation_name) enter image description here please i got this error while running the following code https://github.com/awslabs/realtime-fraud-detection-with-gnn-on-dgl/tree/main/src/sagemaker. once shouted the endpoint and called it from lambda I received that, I redid several times it's always the same error, can you help me please

I run the preprocessing code then train the model on sagemaker , after that i deploy it ( the code is from aws : https://github.com/awslabs/realtime-fraud-detection-with-gnn-on-dgl/tree/main/src/sagemaker) i invoke the endpoint with lambda and i receive this server error, in the first it's was ok then it's became like this , I tried everything but still the same problem,

[amztc34283]

same here, have you solved this?

[simoazzo]

Has anyone resolved this? Thank you

[mateusz-serek-wttech]

In my case, this was caused by trying to serialize a numpy float using json.dumps() in the model entry point inside output_fn

[rromanchuk]

How did you get that stracktrace for the error responsible for the 500 error on invocation error on cloudwatch?

sort of related aws/sagemaker-inference-toolkit#70