文件数量多，KV缓存查询速度慢

[xxWeiDG]

当文件数量多的时候，kv缓存查询的速度降低很多，有什么优化办法吗

[lskeeper]

Nice work! I have a question related to the above one. In case we are dealing with a large scale corpus, it means the KV cache will likely be stored in remote storage, which would incur costs from network communication. In this case, do you still see clear win on inference speed-up?

[xxWeiDG]

It's not just the network cost, the more data you have the slower it will be. It's not just the network cost, the more data you have the slower it will be. Xiaohua Yan ***@***.***>于2024年12月16日 周一12:37写道：

…

Nice work! I have a question related to the above one. In case we are dealing with a large scale corpus, it means the KV cache will likely be stored in remote storage, which would incur costs from network communication. In this case, do you still see clear win on inference speed-up? — Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BBSGQBLFJOFFM4KSCXZNSMD2FZKG7AVCNFSM6AAAAABSUA7WV2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBUGU3TQMZQHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>