[Bug]

[thiagoldaniel]

Priority

P1-Stopper

OS type

Ubuntu

Hardware type

Xeon-other (Please let us know in description)

Installation method

• Pull docker images from hub.docker.com
• Build docker images from source

Deploy method

• Docker compose
• Docker
• Kubernetes
• Helm

Running nodes

Single Node

What's the version?

When running the container I receive this error message when validating the service.
2024-11-24T02:42:50.232633Z INFO download: text_generation_launcher: Starting check and download process for Intel/neural-chat-7b-v3-3
2024-11-24T02:43:02.184459Z ERROR download: text_generation_launcher: Download process was signaled to shutdown with signal 4:

Description

Error on start Docker

Reproduce steps

https://opea-project.github.io/latest/getting-started/README.html

Raw log

2024-11-24T02:42:33.812733Z  INFO hf_hub: Token file not found "/root/.cache/huggingface/token"
2024-11-24T02:42:33.831721Z  INFO text_generation_launcher: Model supports up to 32768 but tgi will now set its default to 4096 instead. This is to save VRAM by refusing large prompts in order to allow more users on the same hardware. You can increase that size using `--max-batch-prefill-tokens=32818 --max-total-tokens=32768 --max-input-tokens=32767`.
2024-11-24T02:42:50.231954Z  WARN text_generation_launcher::gpu: Cannot determine GPU compute capability: AssertionError: Torch not compiled with CUDA enabled
2024-11-24T02:42:50.232133Z  INFO text_generation_launcher: Using attention paged - Prefix caching 0
2024-11-24T02:42:50.232276Z  INFO text_generation_launcher: Default `max_input_tokens` to 4095
2024-11-24T02:42:50.232332Z  INFO text_generation_launcher: Default `max_total_tokens` to 4096
2024-11-24T02:42:50.232360Z  INFO text_generation_launcher: Default `max_batch_prefill_tokens` to 4145
2024-11-24T02:42:50.232633Z  INFO download: text_generation_launcher: Starting check and download process for Intel/neural-chat-7b-v3-3
2024-11-24T02:43:02.184459Z ERROR download: text_generation_launcher: Download process was signaled to shutdown with signal 4:
Error: DownloadError

Attachments

No response

[wangkl2]

@thiagoldaniel I cannot reproduce this issue on my end. May I ask which Xeon product/SKU are you using?