Feature Request: Add OLMoE

[natolambert]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Add this model (and other variants) https://huggingface.co/allenai/OLMoE-1B-7B-0924-Instruct

Motivation

We recently released the OLMoE model at Ai2. 1.3b active / 6.9b total param MoE model. Seems solid, and we'd love people to use it.

Possible Implementation

Should be able to quickly use mix of existing OLMo implementation + Transformers version https://github.com/huggingface/transformers/blob/main/src/transformers/models/olmoe/modeling_olmoe.py

[rkinas]

Voted! It is ideal for mobile solution. Quantized will be even better :)

[natolambert]

I may try this in my free time, but I'm kind of chaotic these days with research + writing so not too optimistic on timeline. I'll also try and get the OLMoE lead on it after VLLM.

[Meshwa428]

Any updates

[foldl]

#9462 is doing this.

And, I have also implemented this in ChatLLM.cpp.

[dirkgr]

Looks like this is done?

[felladrin]

Looks like this is done?

Indeed. Should have been completed by this PR:

• Implement OLMoE architecture #9462

GGUFs are already available on HF

• https://huggingface.co/models?search=olmoe%20gguf

[mseri]

Does this mean that it will be easier to implement compatibility to https://allenai.org/blog/olmo2 ?

[mseri]

Already there: #10500 :)