go-llama.cpp
Inference, closer to the metal. Running a model locally changes the shape of an application. This maintained fork of go-skynet/go-llama.cpp connects Go code to llama.cpp through cgo and native C++ bindings.
The interesting problem.
A native inference engine and a Go application have different expectations about memory, cancellation, configuration, and execution. The boundary deserves as much attention as the model itself.
The approach.
The project exposes Go interfaces for local GGUF model inference, streaming, and related capabilities. Its documentation records setup requirements, supported behavior, and remaining limits.
Inside the project
- Local GGUF model inference
- Streaming and cancellation interfaces
- Chat templates and embeddings
- Explicit native build requirements
This is a maintained fork of go-skynet/go-llama.cpp, built on the work of the llama.cpp community. Model compatibility and performance depend on the version, hardware, and configuration. Explore the upstream project ↗