Local LLM inference

A faux stream that looks roughly like what a real chat completion looks like coming out of the server, slowed down enough to read:

Qwen3.6 is multimodal, so the same OpenAI-compatible endpoint answers questions about images, not just text. Nine callers lean on it, six of them automatically:

• Home Assistant’s voice assistant uses it as the conversation agent through Home Assistant’s built-in LLM API, with plain scripts as its tools.
• The receipt reader sends a photographed shopping docket and gets the lines back as JSON, which is booked into stock only if the totals and item counts add up.
• Paperless gets short titles for scanned pages, and a periodic, read-only second opinion on how each document was filed.
• A finance dashboard, a food log and the grocery catalogue ask it for spending categories, food estimates and nutrition details.

The house rule is that it is never the sole judge: “unknown” is always an accepted answer, a docket books only when its arithmetic closes, and a spending category stays out of every total until a person taps it. How those pieces fit together is on the household stack page. Every one of these model calls stays on the LAN, and there is no per-call bill.

The model is a 35-billion-parameter mixture of experts with about 3 billion active per token, plus a vision encoder. The trick to running it fast on a single 16 GB consumer GPU is being opinionated about which tensors live on which device:

• Attention layers, router, embeddings, the vision encoder and the KV cache live on the RTX 4080, about 8 GiB of VRAM at a 256K context window.
• All 40 layers’ MoE expert weights stay in container RAM, pinned with mlock so the kernel can never page them out. That takes ~25 GiB at Q6_K and never moves.
• Decode lands around 54 tokens/second with q8_0 KV cache and flash-attention on; prefill runs at roughly 730 to 830, depending on prompt length. Prefill is bound by the PCIe bus, because the experts stream from system RAM once per batch, so the physical batch size matters more here than CPU or GPU speed.

# inside CT 108 on the 4080 node (llama.cpp built against CUDA 13)
# systemd unit: /etc/systemd/system/llama-server.service (file names shortened)
# LimitMEMLOCK=infinity   <- required for mlock past the 8 MiB default
# CPUAffinity=0-15        <- the P-cores only: +12% decode, measured
# LD_PRELOAD=libjemalloc.so.2  <- keeps RSS steady with ~25 GiB pinned

llama-server \
  --model /models/Qwen3.6-35B-A3B-Q6_K.gguf \
  --mmproj /models/mmproj-Qwen3.6-35B-A3B-F16.gguf \
  --host 0.0.0.0 --port 11434 \
  --n-gpu-layers 99 \
  --n-cpu-moe 99 \
  --load-mode mlock \
  --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --ctx-size 262144 \
  --batch-size 2048 --ubatch-size 2048 \
  --fit off \
  --parallel 2 --kv-unified \
  --cache-ram 6144 \
  --threads 16 --threads-batch 16 \
  --jinja \
  --reasoning off \
  --temp 0.7 --top-p 0.8 --top-k 20 --presence-penalty 1.5 \
  --metrics

The --mmproj flag loads the vision projector, so the one server handles text, images and short video. Two request slots share one unified 256K-token KV pool (--parallel 2 --kv-unified), so a single request can still use the whole context. The prompt cache is capped with --cache-ram 6144: a new build’s 8 GiB default was bigger than the container’s headroom and caused a run of out-of-memory kills until a 1 GiB cap went in. Once the container got more room the cap was raised to 6 GiB, and the defaults get re-checked after every rebuild. The LXC config gets lxc.prlimit.memlock: unlimited so the in-container LimitMEMLOCK isn’t silently clipped by the host’s 8 MiB default.

The 4080 does more than serve the LLM. Two other containers share it: an XTTS voice container at roughly 3 GiB and a Whisper speech-to-text container at about 2.1 GiB, both feeding the local voice stack. With the model and its vision encoder sitting near 8 GiB, that packs a full local AI stack (chat, vision, TTS and STT) onto one 16 GiB consumer GPU with about 3 GiB to spare at idle. Any tuning change that would leave less than 2 GiB free, enough for a transcription and a synthesis at once, gets reverted.

The ZFS ARC was also capped at 8 GiB on the host; its default of half of RAM would have fought with the mlock-pinned expert weights. Cold reads from the NAS pool are slightly slower on first access; everything else benefits from the freed RAM.

llama.cpp’s server speaks both its native completion API and an OpenAI-compatible chat endpoint, so existing OpenAI client libraries point at it with nothing more than a base-URL change. With the projector loaded, that same endpoint takes images. This is the shape of the call the receipt reader makes with a photographed docket:

from openai import OpenAI

client = OpenAI(
    base_url="http://192.168.1.34:11434/v1",
    api_key="not-used",
)

resp = client.chat.completions.create(
    model="qwen3.6",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "List every item line on this shop docket as JSON."},
            {"type": "image_url",
             "image_url": {"url": "data:image/jpeg;base64,..."}},
        ],
    }],
)
print(resp.choices[0].message.content)