ReBASED

[kabachuha]

Model description

Mirroring #29466, the newer model utilizing LinearAttention adds RMS norm to the attention forwarding and contracts the Taylor expansion to include only the third term, showing better performance and like BASED outperforms Mamba and stuff

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

The repo https://github.com/corl-team/rebased

[ArthurZucker]

If there are no model weights, the chances of merging it are very low 😢

[kabachuha]

@ArthurZucker Their team now provided the weights and generation code corl-team/rebased#2

[elephantmipt]

From my perspective, there is a problem with Based/ReBased models in Huggingface Transformers: if someone wants to train/finetune a model, they need to utilize IO-Aware Triton kernels because the vanilla Torch implementation has a significant memory footprint. There is an implementation of several models here, and theoretically, it is possible to include flash_linear_attention as an optional dependency, similar to flash_attention. Additionally, there is still an option to use model for inference without any kernels.