Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Install wasm32-wasi
target.
rustup target add wasm32-wasi
Install WasmEdge + GGML
curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash -s -- -v 0.13.5 --plugins wasi_nn-ggml wasmedge_rustls
bash <(curl -sSfL 'https://raw.githubusercontent.com/LlamaEdge/LlamaEdge/main/run-llm.sh')
Access the chatbot UI from a local web browser:
http://localhost:8080/
bash <(curl -sSfL 'https://raw.githubusercontent.com/LlamaEdge/LlamaEdge/main/run-llm.sh') --interactive
sh <(curl -sSfL 'https://raw.githubusercontent.com/GaiaNet-AI/gaianet-node/main/demo/init.sh')
sh <(curl -sSfL 'https://raw.githubusercontent.com/GaiaNet-AI/gaianet-node/main/demo/start.sh')
Load the browser at http://localhost:8080/
Or, try the API endpoint:
curl -X POST http://localhost:8080/v1/chat/completions \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"messages":[{"role":"system", "content": "You are a helpful assistant."}, {"role":"user", "content": "Where is Paris?"}], "model":"Llama-2-7b-chat-hf-Q5_K_M"}'
Checkout the file ~/gaianet/start-log.txt
to see the prompt supplementation and manipulation.
sh <(curl -sSfL 'https://raw.githubusercontent.com/GaiaNet-AI/gaianet-node/main/demo/stop.sh')
git clone https://github.com/second-state/WasmEdge-WASINN-examples
cd WasmEdge-WASINN-examples
Build the basic example
cd wasmedge-ggml/basic
cargo build --target wasm32-wasi --release
cp target/wasm32-wasi/release/wasmedge-ggml-basic.wasm .
Download a non-chat model
curl -LO https://huggingface.co/second-state/StarCoder2-7B-GGUF/resolve/main/starcoder2-7b-Q5_K_M.gguf
Run it!
wasmedge --dir .:. \
--env n_predict=100 \
--nn-preload default:GGML:AUTO:starcoder2-7b-Q5_K_M.gguf \
wasmedge-ggml-basic.wasm default
Try a few examples.
USER:
def print_hello_world():
USER:
fn is_prime(n: u64) -> bool {
USER:
Write a Rust function to check if an input number is prime:
Build a chat example
cd wasmedge-ggml/llama
cargo build --target wasm32-wasi --release
cp target/wasm32-wasi/release/wasmedge-ggml-llama.wasm .
Copy it to another device with a different GPU!
scp -i YOUR-KEY.pem wasmedge-ggml-llama.wasm remoteuser@ip-addr:/home/remoteuser
SSH into the remote machine above and download a chat model.
curl -LO https://huggingface.co/second-state/Llama-2-7B-Chat-GGUF/resolve/main/Llama-2-7b-chat-hf-Q5_K_M.gguf
Run it!
wasmedge --dir .:. \
--nn-preload default:GGML:AUTO:Llama-2-7b-chat-hf-Q5_K_M.gguf \
wasmedge-ggml-llama.wasm default