Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add script to wait for the model to appear when using KServe Modelcar #141

Closed
wants to merge 1 commit into from
Closed

Add script to wait for the model to appear when using KServe Modelcar #141

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions Dockerfile.ubi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,9 +196,11 @@ RUN umask 002 \
&& chmod g+rwx $HOME /usr/src /workspace

COPY LICENSE /licenses/vllm.md
COPY --chown=2000:0 --chmod=554 extras/wait-modelcar.sh .

USER 2000
ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server"]
ENTRYPOINT ["/workspace/wait-modelcar.sh"]
CMD ["python3", "-m", "vllm.entrypoints.openai.api_server"]


FROM vllm-openai as vllm-grpc-adapter
Expand All @@ -217,4 +219,4 @@ ENV GRPC_PORT=8033 \
DISABLE_LOGPROBS_DURING_SPEC_DECODING=false

USER 2000
ENTRYPOINT ["python3", "-m", "vllm_tgis_adapter", "--uvicorn-log-level=warning"]
CMD ["python3", "-m", "vllm_tgis_adapter", "--uvicorn-log-level=warning"]
14 changes: 14 additions & 0 deletions extras/wait-modelcar.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/bin/bash

if [ "${MODEL_INIT_MODE}" = "async" ] ; then
echo "Waiting for model files (modelcar) to be present..."
until test -e /mnt/models; do
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we have a (configurable) timeout here?

sleep 1
done

echo "Model files are now available."
fi

echo "Starting model server..."
eval $@

Loading