Add script to wait for the model to appear when using KServe Modelcar

When using OCI containers for model storage in KServe (modelcar), there
is the possibility that the model server starts before the model has
been fully downloaded. When this happens, the model server would
terminate with error because the model path is empty.

This adds a small script to wait for the cluster to fully download the
model container before invoking the model server. The waiting is
triggered when the MODELCAR_PATH environment variable is set to the path
where the model should appear (which should be the same path for loading
the model).

Signed-off-by: Edgar Hernández <[email protected]>

Dockerfile.ubi

@@ -196,9 +196,11 @@ RUN umask 002 \
     && chmod g+rwx $HOME /usr/src /workspace
 
 COPY LICENSE /licenses/vllm.md
+COPY --chown=2000:0 --chmod=554 extras/wait-modelcar.sh .
 
 USER 2000
-ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server"]
+ENTRYPOINT ["/workspace/wait-modelcar.sh"]
+CMD ["python3", "-m", "vllm.entrypoints.openai.api_server"]
 
 
 FROM vllm-openai as vllm-grpc-adapter
@@ -217,4 +219,4 @@ ENV GRPC_PORT=8033 \
     DISABLE_LOGPROBS_DURING_SPEC_DECODING=false
 
 USER 2000
-ENTRYPOINT ["python3", "-m", "vllm_tgis_adapter", "--uvicorn-log-level=warning"]
+CMD ["python3", "-m", "vllm_tgis_adapter", "--uvicorn-log-level=warning"]

extras/wait-modelcar.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+if [ ! -z "${MODELCAR_PATH}" ] ; then
+  echo "Waiting for model files (modelcar) to be present..."
+  until test -e "${MODELCAR_PATH}"; do
+    sleep 1
+  done
+
+  echo "Model files are now available."
+fi
+
+echo "Starting model server..."
+eval $@
+