IndexError while conducting GPT-2 Large benchmark for 1k input tokens.

[JoAnnHang]

Hi!

I followed the instructions in python/llm/dev/benchmark/all-in-one to run benchmark for GPT2-large model.
I'm encountering an IndexError while conducting the all-in-one benchmark (test-api: optimized-model) for the GPT-2 Large model for 1024/128 input/output tokens. It works fine with 512 tokens.

• Model: openai-community/gpt2-large
• Input/output tokens: 1024/128
• Error: IndexError: index out of range in self

Any assistance would be greatly appreciated. Thank you.

[lzivan]

Hi @JoAnnHang, we are currently reproducing this issue. And will update here for any updates :)

[lzivan]

Hi @JoAnnHang, we've reproduced the issue, will get back to you once we have a solution :)

[ACupofAir]

Hi!  你好！

I followed the instructions in python/llm/dev/benchmark/all-in-one to run benchmark for GPT2-large model.我按照 python/llm为 GPT2-large 模型运行 benchmark。 I'm encountering an IndexError while conducting the all-in-one benchmark (test-api: optimized-model) for the GPT-2 Large model for 1024/128 input/output tokens. It works fine with 512 tokens.我在为 GPT-2 大型模型执行 1024/128 个输入/输出令牌的一体化基准测试（test-api：optimized-model）时遇到 IndexError。它与 512 个令牌一起工作得很好。

• Model: openai-community/gpt2-large模型： openai-community/gpt2-large
• Input/output tokens: 1024/128输入/输出令牌：1024/128
• Error: IndexError: index out of range in self错误：IndexError：索引超出 self 范围

Any assistance would be greatly appreciated. Thank you.任何帮助将不胜感激。谢谢。

Refer to https://huggingface.co/openai-community/gpt2-large/blob/main/config.json line13 n_ctx=1024 which means that the maximum length of the input + output token is 1024. So 1024/128 will out of range, 512/128 is OK.