Now I use lm-format-forcer+vllm and it takes 10 seconds to generate a json. This speed is not feasible in my current business. Is there any accelerated solution, such as a paper, plan or pr?

[lwdnxu]

Now I use lm-format-forcer+vllm and it takes 10 seconds to generate a json. This speed is not feasible in my current business. Is there any accelerated solution, such as a paper, plan or pr?

[noamgat]

This has been discussed, also in the vLLM repo. There are some profiling efforts going on there, it might have to do with copying logit buffers from CPU to GPU memory. There is no clear cut solution yet, if you want to step in to investigate, we will be very grateful :)

[ckhfor]

python -> cpp, python is too slow...
multithreading, python gil...
pre-compute accept/reject/uncertain
... many ways to accelerate