This is a project that shows you how to run LLM inference and build OpenAI-compatible API services for the Llama2 series of LLMswith Rust and WasmEdge.

• The folder api-server includes the source code and instructions to create OpenAI-compatible API service for your llama2 model or the LLama2 model itself.
• The folder chat includes the source code and instructions to run llama2 models that can have continuous conversations.
• The folder simple includes the source code and instructions to run llama2 models that can answer one question.

The Rust+Wasm stack provides a strong alternative to Python in AI inference.

• Lightweight. The total runtime size is 30MB as opposed to 4GB for Python and 350MB for Ollama.
• Fast. Full native speed on GPUs.
• Portable. Single cross-platform binary on different CPUs, GPUs, and OSes.
• Secure. Sandboxed and isolated execution on untrusted devices.
• Container-ready. Supported in Docker, containerd, Podman, and Kubernetes.

For more information, please check out Fast and Portable Llama2 Inference on the Heterogeneous Edge.

The llama-utils project, in theory, supports all Language Learning Models (LLMs) based on the llama2 framework in GGUF format. Below is a list of models that have been successfully verified to work on both Mac and Jetson Orin platforms. We are committed to continuously expanding this list by verifying additional models. If you have successfully operated other LLMs, don't hesitate to contribute by creating a Pull Request (PR) to help extend this list.

• Llama-2-7B-Chat-GGUF | Guide

• Llama-2-13B-chat-GGUF | Guide

• CodeLlama-13B-GGUF | Guide

• Getting Started with Orca-2-13B Guide

• Mistral-7B-Instruct-v0.1-GGUF | Guide

• MistralLite-7B-GGUF | Guide

• WizardCoder-Python-7B | Guide

• OpenChat-3.5-GGUF | Guide

• BELLE-Llama2-13B-Chat-0.4M-GGUF

• Yi-34B | Guide

• wizard-vicuna-13B-GGUF | Guide

• Zephyr-7B | Guide

• CausalLM-14B | Guide

• TinyLlama-1.1B-Chat-v0.3 | Guide

• Baichuan2-7B-Chat-GGUF

• Baichuan2-13B-Chat-GGUF | Guide

• OpenHermes-2.5-Mistral-7B-GGUF

• Dolphin-2.2-Yi-34B-GGUF | Guide

• rpguild-chatml-13B-GGUF

• CodeShell-7B-Chat-GGUF

Install WasmEdge 0.13.5+WASI-NN ggml plugin(Metal enabled on apple silicon) via installer

curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash -s -- --plugin wasi_nn-ggml
# After install the wasmedge, you have to activate the environment.
# Assuming you use zsh (the default shell on macOS), you will need to run the following command
source $HOME/.zshenv

The installer from WasmEdge 0.13.5 will detect cuda automatically.

If CUDA is detected, the installer will always attempt to install a CUDA-enabled version of the plugin.

Install WasmEdge 0.13.5+WASI-NN ggml plugin via installer

curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash -s -- --plugin wasi_nn-ggml
# After installing the wasmedge, you have to activate the environment.
# Assuming you use bash (the default shell on Ubuntu), you will need to run the following command
source $HOME/.bashrc

This version is verified on the following platforms:

1. Nvidia Jetson AGX Orin 64GB developer kit
2. Intel i7-10700 + Nvidia GTX 1080 8G GPU
3. AWS EC2 g5.xlarge + Nvidia A10G 24G GPU + Amazon deep learning base Ubuntu 20.04

If the CPU is the only available hardware on your machine, the installer will install the OpenBLAS version of the plugin instead.

You may need to install libopenblas-dev by apt update && apt install -y libopenblas-dev.

Install WasmEdge 0.13.5+WASI-NN ggml plugin via installer

apt update && apt install -y libopenblas-dev # You may need sudo if the user is not root.
curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash -s -- --plugin wasi_nn-ggml
# After installing the wasmedge, you have to activate the environment.
# Assuming you use bash (the default shell on Ubuntu), you will need to run the following command
source $HOME/.bashrc

Install WasmEdge 0.13.5+WASI-NN ggml plugin via installer

curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash -s -- --plugin wasi_nn-ggml
# After install the wasmedge, you have to activate the environment.
# Assuming you use bash (the default shell on Ubuntu), you will need to run the following command
source $HOME/.bashrc

• After running apt update && apt install -y libopenblas-dev, you may encounter the following error:

  ...
  E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
  E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?

  This indicates that you are not logged in as root. Please try installing again using the sudo command:

  sudo apt update && sudo apt install -y libopenblas-dev

• After running the wasmedge command, you may receive the following error:

  [2023-10-02 14:30:31.227] [error] loading failed: invalid path, Code: 0x20
  [2023-10-02 14:30:31.227] [error]     load library failed:libblas.so.3: cannot open shared object file: No such file or directory
  [2023-10-02 14:30:31.227] [error] loading failed: invalid path, Code: 0x20
  [2023-10-02 14:30:31.227] [error]     load library failed:libblas.so.3: cannot open shared object file: No such file or directory
  unknown option: nn-preload

  This suggests that your plugin installation was not successful. To resolve this issue, please attempt to install your desired plugin again.

• After executing the wasmedge command, you might encounter the error message: [WASI-NN] GGML backend: Error: unable to init model. This error signifies that the model setup was not successful. To resolve this issue, please verify the following:

  1. Check if your model file and the WASM application are located in the same directory. The WasmEdge runtime requires them to be in the same location to locate the model file correctly.

  2. Ensure that the model has been downloaded successfully. You can use the command shasum -a 256 <gguf-filename> to verify the model's sha256sum. Compare your result with the correct sha256sum available on the Hugging Face page for the model.

     

The WASI-NN ggml plugin embedded llama.cpp as its backend.