Bonsai 2 9B

· llama.cpp · 32K context · RTX 3080 · 12 GB

Prism ML's ternary Bonsai 2 9B, based on Qwen3.5-9B, with thinking and tool calling. The Q2_0 packing requires upstream llama.cpp with Bonsai Hadamard support.

155 tok/sdecode

2,654 tok/sprefill

32Kcontext window

Tested on this cardOct 16, 2026

Run it

Follow the steps below to download and run this recipe. Omarchy Local AI lists the recipes it can launch on your hardware. The server then answers on http://localhost:8000/v1.

1 Download the weights

hf download prism-ml/Ternary-Bonsai-2-9B-gguf Ternary-Bonsai-2-9B-Q2_0.gguf \
  --revision 5b2abb02362efbed46bbe204014b02837778b2c2 \
  --local-dir ~/models/Ternary-Bonsai-2-9B-gguf-5b2abb02

2 Start the server

docker run --rm \
  --gpus all \
  -p 8000:8080 \
  --shm-size 1g \
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -v ~/models/Ternary-Bonsai-2-9B-gguf-5b2abb02:/models:ro \
  --entrypoint /app/llama-server \
  ghcr.io/ggml-org/llama.cpp:server-cuda12@sha256:53bec65a6120e8ab8cb76af725c3b05a7f098b5308c96e5d9f8dad98f1f1f639 \
  --model /models/Ternary-Bonsai-2-9B-Q2_0.gguf \
  --alias Bonsai-2-9B \
  --host 0.0.0.0 \
  --port 8080 \
  --n-gpu-layers 999 \
  --split-mode none \
  --ctx-size 32768 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --jinja \
  --metrics \
  --no-mmproj \
  --fit off \
  --batch-size 512 \
  --ubatch-size 128 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0 \
  --presence-penalty 1.0 \
  --repeat-penalty 1.0 \
  --repeat-last-n 64

What it passed

✓ loadLoads and serves within an hour

✓ chatAnswers a plain question and stops on its own

✓ reasoningThinks separately and gets 17 × 23 right

✓ toolsCalls a tool with the right arguments and uses the result

✓ contextRecalls a code buried in a prompt that fills 85% of the window

✓ speedDecodes at 15 tok/s or more

Passed all six checks on a real NVIDIA GeForce RTX 3080 (12 GiB) on Oct 16, 2026.

Details

Weights: prism-ml/Ternary-Bonsai-2-9B-gguf @ 5b2abb0236 ›
Image: ghcr.io/ggml-org/llama.cpp:server-cuda12@sha256:53bec65a6120e8ab8cb76af725c3b05a7f098b5308c96e5d9f8dad98f1f1f639
Launch: llama.cpp-bonsai-2-9b-q2-0-32k-cuda12 ›
Recipe file: registry/recipes/nvidia/rtx-3080-12gb/bonsai-2-9b.llama.cpp.32k.json ›
Model released: Sep 23, 2026
Tested: owner, NVIDIA GeForce RTX 3080 (12 GiB)