Long documents

[timsuchanek]

Would it be possible to summarize documents with length > 758 tokens?
Using https://github.com/allenai/longformer could be interesting for that use-case.

[jiacheng-xu]

Hi! Thanks for your suggestion. Longformer is great for the long document scenario.
In this project, I can actually change the max_len to any, see

DiscoBERT/data_preparation/data_structure.py

Line 20 in a96922a

self.tokenizer.max_len = 768

What I do is randomly initializing the extended part and fine-tuning on the downstream tasks. You can also change the num 768 to any number you want.