Why llamaCpp only support batchsize=1 in text generation

[neozhang307]

I wonder why there is such a limitation in llamacpp?
if self.config.task == "text-generation":
if input_shapes["batch_size"] != 1:
raise ValueError("Batch size must be 1 for LlamaCpp text generation")

In '/optimum_benchmark/backends/llama_cpp/backend.py'

[IlyasMoutawwakil]

@baptistecolle

[baptistecolle]

Hi,

Llama.cpp supports batch inference. However, we use a Python binding (llama-cpp-python) to interact with it, which currently does not support batch inference abetlen/llama-cpp-python#771.

Once this feature is added, we can remove the current restriction and start benchmarking with batches

[neozhang307]

OK