[Usage]: vllm context length handling method

[whoo9112]

Your current environment

I am writing because I have a question during the test using vllm.
I did RAG with gemma2 model using vllm in a100 40gb.

We have seen here that the fragments of the document searched sometimes lead to prompts entering gemma2 going beyond the 8192 token max contextlength.

But it still generates the answer. In this regard, may I know which function or method is responsible for handling max context length?

How would you like to use vllm

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.