(Have PR) Speed up BeamScorer to make GPT-2 generation 2-3x faster

[fzyzcjy]

Feature request

Please refer to #21234 (comment) for details (I create a new issue because I guess the reply to ancient issue will not be seen)

Motivation

Your contribution

[amyeroberts]

cc @gante

[gante]

Hi @fzyzcjy 👋

The reason for not merging your suggested PR is still valid, we are currently refactoring generate and beam_search will look very different from what it is at the moment when we're done :)

[fzyzcjy]

I see. May I know when will the refactor be done? (seems many months have passed...) and looking forward to the new code!

[gante]

@fzyzcjy You can see the tracker here 🤗

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.