KoboldCpp

OpenClaw + KoboldCpp Integration

KoboldCpp is one of the most popular local LLM runtimes — efficient, cross-platform, and compatible with virtually every GGUF model. When combined with OpenClaw, you get a fully local AI assistant: no API costs, no data leaving your machine, and offline capability.

This is the setup for people who are serious about privacy, want to experiment with local models, or simply don't want to pay per-token forever.

Why KoboldCpp + OpenClaw?

Zero API Costs: Once you download a model, inference is free. Run it 24/7 without paying per token.

Complete Privacy: Nothing leaves your machine. Your conversations, your files, your data — all local.

Offline Operation: Works without internet (for the AI itself — OpenClaw's other features that call external APIs are separate).

Model Freedom: Run Llama 3.3, Mistral, Qwen 2.5, DeepSeek, Phi-4, Gemma, or any GGUF-format model.

Hardware Flexibility: Works on CPU (slow but functional), NVIDIA GPU (CUDA), AMD GPU (ROCm), and Apple Silicon (Metal).

Recommended Hardware

For OpenClaw as a daily assistant, 7B Q4 models on Apple Silicon are the sweet spot — fast responses, good quality, zero cost.

Recommended Models

For General Assistant Use

• Llama 3.3 70B Q4_K_M (if you have the GPU RAM — best quality)
• Llama 3.1 8B Q4_K_M (fast, great quality for size)
• Qwen 2.5 7B Q4_K_M (excellent multilingual)
• Mistral 7B Q4_K_M (fast, reliable)

For Coding Tasks

• DeepSeek Coder V2 Lite Q4_K_M
• Qwen 2.5 Coder 7B Q4_K_M

For Privacy-Sensitive Use

• Phi-4 Q4_K_M (Microsoft, compact but capable)
• Gemma 2 9B Q4_K_M (Google, good instruction following)

All available on Hugging Face in GGUF format.

Step-by-Step Setup

Step 1: Install KoboldCpp

macOS (Apple Silicon — recommended):

brew install koboldcpp

Or download the pre-built binary from github.com/LostRuins/koboldcpp.

Windows: Download koboldcpp.exe from the releases page. Run directly — no installation needed.

Linux:

git clone https://github.com/LostRuins/koboldcppcd koboldcppmake LLAMA_CUBLAS=1  # for NVIDIA GPU# or: make LLAMA_METAL=1  # for Apple Silicon

Step 2: Download a Model

# Install Hugging Face CLIpip install huggingface-hub# Download Llama 3.1 8B Q4 (recommended starter)huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \  Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \  --local-dir ~/models

Or browse huggingface.co/models?library=gguf and download manually.

Step 3: Start KoboldCpp

koboldcpp \  --model ~/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \  --port 5001 \  --contextsize 8192 \  --gpulayers 99 \  --host 127.0.0.1

Key flags:

• --gpulayers 99: Offload all layers to GPU (Metal on Apple Silicon, CUDA on NVIDIA)
• --contextsize: Context window. 8192 is a good default; go higher if your RAM allows
• --host 127.0.0.1: Keep it local only (security)

Step 4: Configure OpenClaw

KoboldCpp exposes an OpenAI-compatible API. Configure OpenClaw to use it:

providers:  local:    type: openai-compatible    baseUrl: http://127.0.0.1:5001/v1    apiKey: "koboldcpp"  # any string — KoboldCpp doesn't validate this    model: koboldcpp  # or the model name KoboldCpp reports    contextLength: 8192# Set as default providerdefaultProvider: local

Step 5: Test

Ask OpenClaw anything. If KoboldCpp is running and configured correctly, responses will come from your local model.

openclaw chat "Hello, who are you?"

You should see a response from the local model (not Claude).

Performance Tuning

Apple Silicon (Recommended)

Apple Silicon has unified memory — even the base M1 Mac Mini with 8GB can run 7B models well:

koboldcpp \  --model ~/models/llama-3.1-8b-q4_k_m.gguf \  --port 5001 \  --gpulayers 99 \  --contextsize 8192 \  --usemmap

NVIDIA GPU (CUDA)

koboldcpp \  --model ~/models/llama-3.1-8b-q4_k_m.gguf \  --port 5001 \  --gpulayers 99 \  --usecublas

CPU Only (Fallback)

koboldcpp \  --model ~/models/phi-4-q4_k_m.gguf \  --port 5001 \  --threads 8

Expect 2–10 tokens/second on CPU depending on hardware.

Running KoboldCpp as a Service

To keep KoboldCpp running in the background on macOS:

Create ~/Library/LaunchAgents/koboldcpp.plist:

<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict>  <key>Label</key>  <string>koboldcpp</string>  <key>ProgramArguments</key>  <array>    <string>/usr/local/bin/koboldcpp</string>    <string>--model</string>    <string>/Users/yourname/models/llama-3.1-8b-q4_k_m.gguf</string>    <string>--port</string>    <string>5001</string>    <string>--gpulayers</string>    <string>99</string>  </array>  <key>RunAtLoad</key>  <true/>  <key>KeepAlive</key>  <true/></dict></plist>

Load it:

launchctl load ~/Library/LaunchAgents/koboldcpp.plist

Mixing Local + Cloud

You don't have to go all-local. OpenClaw supports routing different tasks to different providers:

providers:  local:    type: openai-compatible    baseUrl: http://127.0.0.1:5001/v1  claude:    type: anthropic    apiKey: "sk-ant-..."# Use local for most things, Claude for complex tasksrouting:  default: local  complex: claude  # triggered by skill or manually

Privacy Notes

With KoboldCpp as your provider:

• No data sent externally for AI inference
• OpenClaw skills that call external APIs (email, calendar) still make those calls
• Your conversation history stays on your machine
• No usage logs sent to any AI company

Troubleshooting

Slow responses on CPU

• Use a smaller or more quantized model (Q3_K_S instead of Q4_K_M)
• Increase thread count: --threads $(nproc)
• Consider upgrading to a GPU or Apple Silicon machine

Out of memory errors

• Use a more aggressive quantization (Q3 instead of Q4)
• Reduce context size: --contextsize 4096
• Try a smaller model (7B instead of 13B)

OpenClaw can't connect to KoboldCpp

• Verify KoboldCpp is running: curl http://127.0.0.1:5001/v1/models
• Check the port in your OpenClaw config matches
• Ensure --host 127.0.0.1 (or 0.0.0.0 if running on a separate machine)

Model produces poor quality output

• Try a higher quantization (Q4_K_M or Q5_K_M)
• Use a model fine-tuned for instruction following (look for -Instruct in the name)
• Increase context size if conversations feel truncated