From d75c7383445f828eacb691d942fe3e0c1eeb6aaa Mon Sep 17 00:00:00 2001 From: dm4 Date: Wed, 13 Nov 2024 15:20:52 +0800 Subject: [PATCH] [Docs] ggml: add TriLM and BitNet documents in basic/README.md Signed-off-by: dm4 --- wasmedge-ggml/basic/README.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/wasmedge-ggml/basic/README.md b/wasmedge-ggml/basic/README.md index ac0a494..ea7802a 100644 --- a/wasmedge-ggml/basic/README.md +++ b/wasmedge-ggml/basic/README.md @@ -82,3 +82,26 @@ $ wasmedge --dir .:. \ --nn-preload default:GGML:AUTO:grok-1-Q2_K-split-00001-of-00009.gguf \ wasmedge-ggml-basic.wasm default 'hello' ``` + +## TriLM & BitNet Models + +After the following pull requests are merged, the `TriLM` and `BitNet` models will be supported by the `ggml` plugin with model type `TQ1_0` and `TQ2_0`: +- https://github.com/ggerganov/llama.cpp/pull/7931 +- https://github.com/ggerganov/llama.cpp/pull/8151 + +### Get the Model + +Download the `TriLM` model: + +```bash +curl -LO https://huggingface.co/Green-Sky/TriLM_3.9B-GGUF/resolve/main/TriLM_3.9B_Unpacked-4.0B-TQ2_0.gguf +``` + +### Execute + +```console +$ wasmedge --dir .:. \ + --env n_predict=100 \ + --nn-preload default:GGML:AUTO:TriLM_3.9B_Unpacked-4.0B-TQ2_0.gguf \ + wasmedge-ggml-basic.wasm default +```