[Feature] Upgrade llama.cpp to support Phi-3-mini-128k-instruct and IBM Granite #2668
Labels
enhancement
New feature or request
Comments
|
This is being worked on and we'll have a new release out soon that will have latest llama.cpp |
ThiloteE
changed the title
|
llama.cpp has been upgraded, hence will close this issue.
|
ThiloteE
added a commit
that referenced
this issue
Aug 3, 2024
Resolves #2668 Adds model support for [Phi-3-mini-128k-instruct](https://huggingface.co/GPT4All-Community/Phi-3-mini-128k-instruct) ### Description of Model At the date of writing, the model has strong results in benchmarks (for its parameter size). It claims to support a context of up to 128K. - The model was trained/finetuned on English - License: MIT ### Personal Impression: For 3.8 billion parameters, the model has reasonable output. It is possible to converse and follow tasks. I have held a conversation that held 24k characters and even at that long of a context, it still was able to answer "what is 2x2?" correctly, albeit the responses understandably slightly degrade at that context size. I have seen refusals when it was tasked with certain things and it seems to be finetuned with a particular alignment. Its long context and quality of responses makes it a good model, if you can bear its alignment or your use case happens to fall within the originally intended use cases of the model. It mainly will appeal to English speaking users. ### Critique: This model does not support Grouped Query Attention, that means other models that support GQA may need less RAM/VRAM for the same amount of tokens in the context window. It has been claimed that llama-3-8b (which supports GQA) needs less RAM after a certain point (\~ 8k context). ### Motivation for this pull-request - The model is small and fits into 3GB of VRAM or 4GB of RAM respectively (I set 8GB of RAM as minimum, as the Operating System and other Apps also need some) - The model claims long context and it delivers (although with high RAM usage in longer conversations). - AFAIK, apart from the Qwen1.5 and Qwen2 model family, this is the only generic purpose model family below 4B parameters that delivers that large of a context window and that is also compatible with GPT4All - For it's size it is high on the huggingface open leaderboard benchmark - Made by Microsoft, the model has a reputation - Users were asking for this model ## Checklist before requesting a review - [x] I have performed a self-review of my code. - [ ] If it is a core feature, I have added thorough tests. - [ ] I have added thorough documentation for my code. - [x] I have tagged PR with relevant project labels. I acknowledge that a PR without labels may be dismissed. - [ ] If this PR addresses a bug, I have provided both a screenshot/video of the original bug and the working solution. Signed-off-by: ThiloteE <[email protected]>
10 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Feature Request
It would be nice if llama.cpp could be upgraded because there are new models supported in the recent versions, e.g.:
IBM Granite (https://github.com/ibm-granite/granite-code-models), see llama: extend for small granite models ggerganov/llama.cpp#7481, integrated into llama.cpp on May 28. Model is available here: https://huggingface.co/ibm-granite/granite-8b-code-instruct-GGUF
Phi-3-medium-128k, see Add phi3 128K model support ggerganov/llama.cpp#7225, seems to be integraed end of May. Model is available here: https://huggingface.co/bartowski/Phi-3-medium-128k-instruct-GGUF
The text was updated successfully, but these errors were encountered: