Simple setup for llama.cpp on macos

refarer,Mon Feb 19 2024•llama-cpp

Step 1

Clone llama.cpp

git clone https://github.com/ggerganov/llama.cpp

Build llama.cpp

cd llama.cpp
make

Download a model

cd models
curl -O https://huggingface.co/TheBloke/stablelm-zephyr-3b-GGUF/blob/main/stablelm-zephyr-3b.Q5_K_M.gguf

Edit .zshrc

vi ~/.zshrc

Add this function to the end of the file

function llm() {
        $HOME/llama.cpp/main 2>/dev/null -m $HOME/llama.cpp/models/stablelm-zephyr-3b.Q5_K_M.gguf --color -c 4096 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "<|user|>\n{$1}<|endoftext|>\n<|assistant|>"
}

Reload your zsh config

source ~/.zshrc

Run commands like so:

llm "js code for printing todays date"

NOTE: If you don't clone llama.cpp in your home directory you will have to update the paths above