llama.cpp

Roadmap / Project status / Manifesto / ggml

Inference of Meta's LLaMA model (and others) in pure C/C++

Recent API changes

• Changelog for libllama API
• Changelog for llama-server REST API

Hot topics

• 🔥 Multimodal support arrived in llama-server: #12898 | documentation
• GGML developer experience survey (organized and reviewed by NVIDIA): link
• A new binary llama-mtmd-cli is introduced to replace llava-cli, minicpmv-cli, gemma3-cli (#13012) and qwen2vl-cli (#13141), libllava will be deprecated
• VS Code extension for FIM completions: https://github.com/ggml-org/llama.vscode
• Universal tool call support in llama-server https://github.com/ggml-org/llama.cpp/pull/9639
• Vim/Neovim plugin for FIM completions: https://github.com/ggml-org/llama.vim
• Introducing GGUF-my-LoRA https://github.com/ggml-org/llama.cpp/discussions/10123
• Hugging Face Inference Endpoints now support GGUF out of the box! https://github.com/ggml-org/llama.cpp/discussions/9669
• Hugging Face GGUF editor: discussion | tool

---

Description

The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is the main playground for developing new features for the ggml library.

Typically finetunes of the base models below are supported as well.

Instructions for adding support for new models: HOWTO-add-model.md

Text-only

• LLaMA 🦙
• LLaMA 2 🦙🦙
• LLaMA 3 🦙🦙🦙
• Mistral 7B
• Mixtral MoE
• DBRX
• Falcon
• Chinese LLaMA / Alpaca and Chinese LLaMA-2 / Alpaca-2
• Vigogne (French)
• BERT
• Koala
• Baichuan 1 & 2 + derivations
• Aquila 1 & 2
• Starcoder models
• Refact

Multimodal

• LLaVA 1.5 models, LLaVA 1.6 models
• BakLLaVA
• Obsidian
• ShareGPT4V
• MobileVLM 1.7B/3B models
• Yi-VL
• Mini CPM
• Moondream
• Bunny
• GLM-EDGE
• Qwen2-VL

• Python: abetlen/llama-cpp-python
• Go: go-skynet/go-llama.cpp
• Node.js: withcatai/node-llama-cpp
• JS/TS (llama.cpp server client): lgrammel/modelfusion
• JS/TS (Programmable Prompt Engine CLI): offline-ai/cli
• JavaScript/Wasm (works in browser): tangledgroup/llama-cpp-wasm
• Typescript/Wasm (nicer API, available on npm): ngxson/wllama
• Ruby: yoshoku/llama_cpp.rb
• Rust (more features): edgenai/llama_cpp-rs
• Rust (nicer API): mdrokz/rust-llama.cpp
• Rust (more direct bindings): utilityai/llama-cpp-rs
• Rust (automated build from crates.io): ShelbyJenkins/llm_client
• C#/.NET: SciSharp/LLamaSharp
• C#/VB.NET (more features - community license): LM-Kit.NET
• Scala 3: donderom/llm4s
• Clojure: phronmophobic/llama.clj
• React Native: mybigday/llama.rn
• Java: kherud/java-llama.cpp
• Zig: deins/llama.cpp.zig

(to have a project listed here, it should clearly state that it depends on llama.cpp)

• AI Sublime Text plugin (MIT)
• cztomsik/ava (MIT)
• Dot (GPL)
• eva (MIT)
• iohub/collama (Apache-2.0)
• janhq/jan (AGPL)
• johnbean393/Sidekick (MIT)
• KanTV (Apache-2.0)
• KodiBot (GPL)
• llama.vim (MIT)
• LARS (AGPL)
• Llama Assistant (GPL)
• LLMFarm (MIT)
• LLMUnity