Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Qwen 2.5推理异常 #62

Open
nvgruel123 opened this issue Dec 10, 2024 · 0 comments
Open

Qwen 2.5推理异常 #62

nvgruel123 opened this issue Dec 10, 2024 · 0 comments

Comments

@nvgruel123
Copy link

nvgruel123 commented Dec 10, 2024 •
edited
Loading

环境：

soc环境
transformers: 4.45.2
torch: 2.0.1
LLM-TPU: commit 8d94a2f (HEAD -> main, origin/main, origin/HEAD)
Author: fangz-ai [email protected]
Date: Tue Dec 3 12:11:59 2024 +0800
tpu-milr: tpu-mlir_b7f199c8e-20241119
driver版本：0.5.1
libsophon: #1 SMP Wed May 22 10:11:21 CST 2024

bm_version

SophonSDK version: 23.09 LTS
sophon-soc-libsophon : 0.5.1
sophon-soc-libsophon-dev : 0.5.1
sophon-mw-soc-sophon-ffmpeg : 0.10.0
sophon-mw-soc-sophon-opencv : 0.10.0
BL2 v2.7(release): Built : 14:05:58, Jun  7 2024
BL31 v2.7(release): Built : 14:05:58, Jun  7 2024
U-Boot 2022.10 (Jun 07 2024 - 14:05:56 +0800) Sophon BM1684X
KernelVersion : Linux aibox-1684x 5.4.217-bm1684 #1 SMP Wed May 22 10:11:21 CST 2024 aarch64 aarch64 aarch64 GNU/Linux
HWVersion: 0x00
MCUVersion: 0x02

模型为通过以下命令转换

./run_compile.sh --model_name qwen2.5-3b --model_path Qwen2.5-3B-Instruct/ --seq_length 2048 --tpu_mlir_path /workspace/LLM-TPU/tpu-mlir_b7f199c8e-20241119/

具体表现为FireflyChat发送“你好”无返回
image

FireflyChat日志如下：

Dec 10 14:49:49 aibox-1684x run.sh[14668]: 2024-12-10 14:49:49,415 [INFO] [base_model.py:556] 用户的输入为：你好
Dec 10 14:49:49 aibox-1684x run.sh[14669]: INFO:     127.0.0.1:42518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Dec 10 14:50:11 aibox-1684x run.sh[14669]: *** bmruntime trace: ***
Dec 10 14:50:11 aibox-1684x run.sh[14669]: ============ check coeff =============
Dec 10 14:50:20 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[07A9AB92], addr[0x124100000], size[0x25180000], Check:**FAILED**
Dec 10 14:50:21 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[0B6836C5], addr[0x18afe5000], size[0x2937000]
Dec 10 14:50:22 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[0B690019], addr[0x166260000], size[0x2937000]
Dec 10 14:50:22 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[1EE520A3], addr[0x1b581c000], size[0x2937000]
Dec 10 14:50:23 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[24204035], addr[0x18dd3e000], size[0x2937000]
Dec 10 14:50:24 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[256846F9], addr[0x19bffb000], size[0x2937000]
Dec 10 14:50:24 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[2A729108], addr[0x1717c4000], size[0x2937000]
Dec 10 14:50:25 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[2B1ED07C], addr[0x1be027000], size[0x2937000]
Dec 10 14:50:25 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[2D9B8BE8], addr[0x19ed54000], size[0x2937000]
Dec 10 14:50:35 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[31C13D6A], addr[0x1c3ad9000], size[0x26e70000], Check:**FAILED**
Dec 10 14:50:36 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[37AD1654], addr[0x185533000], size[0x2937000]
Dec 10 14:50:37 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[60874B61], addr[0x1b2ac3000], size[0x2937000]
Dec 10 14:50:37 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[64F78557], addr[0x16ea6b000], size[0x2937000]
Dec 10 14:50:38 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[6559A27C], addr[0x190a97000], size[0x2937000]
Dec 10 14:50:39 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[6FBB1037], addr[0x1827da000], size[0x2937000]
Dec 10 14:50:39 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[7774CE43], addr[0x1ad011000], size[0x2937000]
Dec 10 14:50:40 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[78FAEA23], addr[0x1c0d80000], size[0x2937000]
Dec 10 14:50:41 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[7D1F4658], addr[0x196549000], size[0x2937000]
Dec 10 14:50:41 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[7E3CD0E6], addr[0x17cd28000], size[0x2937000]
Dec 10 14:50:42 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[91C89E49], addr[0x179fcf000], size[0x2937000]
Dec 10 14:50:42 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[97773FB7], addr[0x153cef000], size[0x2000]
Dec 10 14:50:43 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[995C4FCF], addr[0x1bb2ce000], size[0x2937000]
Dec 10 14:50:43 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[B6ECF8AD], addr[0x18828c000], size[0x2937000]
Dec 10 14:50:44 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[B71A554E], addr[0x17451d000], size[0x2937000]
Dec 10 14:50:45 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[BA2AFA58], addr[0x17fa81000], size[0x2937000]
Dec 10 14:50:45 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[BA308E44], addr[0x163507000], size[0x2937000]
Dec 10 14:50:46 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[BAAA2F16], addr[0x1992a2000], size[0x2937000]
Dec 10 14:50:47 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[BC94AD79], addr[0x1aa2b8000], size[0x2937000]
Dec 10 14:50:47 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[C05F4308], addr[0x16bd12000], size[0x2937000]
Dec 10 14:50:48 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[C1CB27CD], addr[0x1afd6a000], size[0x2937000]
Dec 10 14:50:49 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[CC47430A], addr[0x1a755f000], size[0x2937000]
Dec 10 14:50:51 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[CFF42292], addr[0x149a86000], size[0xa24a000]
Dec 10 14:50:52 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[D018A30F], addr[0x1b8575000], size[0x2937000]
Dec 10 14:50:52 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[D9143144], addr[0x177276000], size[0x2937000]
Dec 10 14:50:53 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[E0160914], addr[0x1607ae000], size[0x2937000]
Dec 10 14:50:54 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[E488ECA0], addr[0x153f53000], size[0x2937000]
Dec 10 14:50:54 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[E76AD548], addr[0x1a4806000], size[0x2937000]
Dec 10 14:50:55 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[E78FDEAF], addr[0x168fb9000], size[0x2937000]
Dec 10 14:50:56 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[FBC7202F], addr[0x1a1aad000], size[0x2937000]
Dec 10 14:50:56 aibox-1684x run.sh[14669]: Coeff, chip[0], SHA[FFF9B566], addr[0x1937f0000], size[0x2937000]
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[embedding] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[embedding_cache] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[lm_head] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[greedy_head] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[penalty_sample_head] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_0] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_0] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_1] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_1] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_2] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_2] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_3] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_3] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_4] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_4] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_5] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_5] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_6] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_6] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_7] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_7] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_8] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_8] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_9] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_9] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_10] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_10] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_11] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_11] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_12] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_12] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_13] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_13] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_14] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_14] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_15] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_15] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_16] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_16] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_17] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_17] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_18] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_18] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_19] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_19] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_20] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_20] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_21] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_21] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_22] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_22] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_23] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_23] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_24] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_24] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_25] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_25] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_26] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_26] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_27] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_27] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_28] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_28] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_29] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_29] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_30] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_30] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_31] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_31] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_32] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_32] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_33] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_33] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_34] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_34] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_35] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ============ check net[block_cache_35] stage[0] =======
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ================
Dec 10 14:50:56 aibox-1684x run.sh[14669]: Total 2 errors
Dec 10 14:50:56 aibox-1684x run.sh[14669]: ERROR:    Exception in ASGI application
Dec 10 14:50:56 aibox-1684x run.sh[14669]: Traceback (most recent call last):
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   File "/usr/local/lib/python3.8/dist-packages/sse_starlette/sse.py", line 284, in __call__
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     await wrap(partial(self.listen_for_disconnect, receive))
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   File "/usr/local/lib/python3.8/dist-packages/sse_starlette/sse.py", line 273, in wrap
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     await func()
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   File "/usr/local/lib/python3.8/dist-packages/sse_starlette/sse.py", line 223, in listen_for_disconnect
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     message = await receive()
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   File "/usr/local/lib/python3.8/dist-packages/uvicorn/protocols/http/h11_impl.py", line 524, in receive
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     await self.message_event.wait()
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   File "/usr/lib/python3.8/asyncio/locks.py", line 309, in wait
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     await fut
Dec 10 14:50:56 aibox-1684x run.sh[14669]: asyncio.exceptions.CancelledError
Dec 10 14:50:56 aibox-1684x run.sh[14669]: During handling of the above exception, another exception occurred:
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   + Exception Group Traceback (most recent call last):
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/uvicorn/protocols/http/h11_impl.py", line 396, in run_asgi
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     result = await app(  # type: ignore[func-returns-value]
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     return await self.app(scope, receive, send)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/fastapi/applications.py", line 1054, in __call__
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     await super().__call__(scope, receive, send)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/starlette/applications.py", line 123, in __call__
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     await self.middleware_stack(scope, receive, send)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/errors.py", line 186, in __call__
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     raise exc
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/errors.py", line 164, in __call__
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     await self.app(scope, receive, _send)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/exceptions.py", line 65, in __call__
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     raise exc
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     await app(scope, receive, sender)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 756, in __call__
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     await self.middleware_stack(scope, receive, send)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 776, in app
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     await route.handle(scope, receive, send)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 297, in handle
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     await self.app(scope, receive, send)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 77, in app
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     raise exc
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     await app(scope, receive, sender)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 75, in app
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     await response(scope, receive, send)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/sse_starlette/sse.py", line 284, in __call__
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     await wrap(partial(self.listen_for_disconnect, receive))
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |   File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   |     raise BaseExceptionGroup(
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:   +-+---------------- 1 ----------------
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     | Traceback (most recent call last):
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     |   File "/usr/local/lib/python3.8/dist-packages/sse_starlette/sse.py", line 273, in wrap
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     |     await func()
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     |   File "/usr/local/lib/python3.8/dist-packages/sse_starlette/sse.py", line 253, in stream_response
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     |     async for data in self.body_iterator:
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     |   File "/firefly_services/FireflyChat/modules/api.py", line 195, in chat_stream_response
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     |     for chunk, status in model_instance.model.stream_inference([m.dict() for m in request.messages]):
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     |   File "/opt/firefly/libfirefly-ai/python/fireflyai/llm/base_model.py", line 65, in stream_inference
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     |     token = self._forward_first(context_tokens)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     |   File "/opt/firefly/libfirefly-ai/python/fireflyai/llm/base_model.py", line 37, in _forward_first
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     |     return self.core.forward_first(tokens)
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     | RuntimeError: BMRuntime internal error.
Dec 10 14:50:56 aibox-1684x run.sh[14669]:     +------------------------------------
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nvgruel123