Is the PPL leaderboard score set with the default mask_length = None? #469
Unanswered
RanchiZhao
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I want to reproduce the PPL task leaderboard, and I need to confirm the default method for PPL:
def get_ppl(self, inputs: List[str], mask_length: Optional[List[int]] = None) -> List[float]:Is the mask_length passed in as None? That is, not considering the length differences between different options?
Beta Was this translation helpful? Give feedback.
All reactions