Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement OLMoE architecture #9462

Merged
merged 1 commit into from
Sep 16, 2024
Merged

Implement OLMoE architecture #9462

merged 1 commit into from
Sep 16, 2024

Conversation

2015aroras
Copy link
Contributor

Implements the recently released open-source OLMoE architecture. Tested with base model, will follow up with instruction tuned model testing. Fixes #9380.

Implementation differences from Qwen2Moe:

  • no shared experts
  • no bias
  • QK norm
  • different hyper-parameters: RMS layer norm eps, number of experts, number of active experts.

Test (base model):

// Download OLMoE weights to models/olmoe-hf
python3 convert_hf_to_gguf.py models/olmoe-hf
./llama-quantize models/olmoe-hf/Olmoe-Hf-64x577M-F16.gguf  models/olmoe-hf/Olmoe-Hf-64x577M-Q4_K_M.gguf Q4_K_M
./llama-cli -m models/olmoe-hf/Olmoe-Hf-64x577M-Q4_K_M.gguf  -p "Bitcoin is" -n 128

Output:

Bitcoin is a new currency that has been created by an unknown person or group of people under the name Satoshi Nakamoto. It is the first decentralized digital currency, and unlike traditional currencies, it is not backed by a government or central authority. This has led to a lot of debate about whether or not Bitcoin is a good investment. In this article, we will explore the pros and cons of Bitcoin and see if it is worth investing in.

What is Bitcoin?

Bitcoin is a digital currency that was created in 2009. It is the first decentralized cryptocurrency, meaning that it is not issued or controlled by a central authority. Bitcoin

@github-actions github-actions bot added the python python script changes label Sep 13, 2024
@foldl foldl mentioned this pull request Sep 13, 2024
Open
4 tasks
@Meshwa428 Meshwa428 mentioned this pull request Sep 13, 2024
Closed
@2015aroras
Copy link
Contributor Author

Various gguf files uploaded to https://huggingface.co/allenai/OLMoE-1B-7B-0924-GGUF/tree/main and https://huggingface.co/allenai/OLMoE-1B-7B-0924-Instruct-GGUF/tree/main (Instruct seems to produce sensible text too). Anything else needed to get this merged?

@2015aroras 2015aroras mentioned this pull request Sep 13, 2024
Open
@Meshwa428
Copy link

@ggerganov why isn't this merged yet?
Any problems?

Please merge it 🙏🏻
So that we ollama folks can use the model in ollama.

@ggerganov ggerganov merged commit 0aadac1 into ggerganov:master Sep 16, 2024
54 checks passed
@ThiloteE ThiloteE mentioned this pull request Oct 8, 2024
Open
dsx1986 pushed a commit to dsx1986/llama.cpp that referenced this pull request Oct 29, 2024
@2015aroras @dsx1986
llama : support OLMoE (ggerganov#9462)
dda150d
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Nov 15, 2024
@2015aroras @arthw
llama : support OLMoE (ggerganov#9462)
0ad970b
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Nov 18, 2024
@2015aroras @arthw
llama : support OLMoE (ggerganov#9462)
c3128c0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ggerganov ggerganov ggerganov approved these changes

Assignees
No one assigned
Labels
python python script changes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add support for OLMoE-1B-7B / 7B
3 participants
@2015aroras @Meshwa428 @ggerganov