Welp-35B-A3B

· llama.cpp · 256K context · RTX 5060 Ti · 16 GB

Quaedra's 13.2 GB GGUF of Qwen3.6-35B-A3B: the routed experts quantized block-wise with GPTQ error feedback in standard llama.cpp formats, so the full 262k context fits on a 16 GB card.

61 tok/sdecode

751 tok/sprefill

256Kcontext window

Tested on this cardOct 13, 2026

Run it

Follow the steps below to download and run this recipe. Omarchy Local AI lists the recipes it can launch on your hardware. The server then answers on http://localhost:8000/v1.

1 Download the weights

hf download quaedra/Welp-35B-A3B-GGUF Welp-35B-A3B.gguf \
  --revision 3e4a3adbda8df07d2889978170d9955c2e59fb7a \
  --local-dir ~/models/Welp-35B-A3B-GGUF-3e4a3adb

2 Start the server

docker run --rm \
  --gpus all \
  -p 8000:8080 \
  --shm-size 16g \
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -v ~/models/Welp-35B-A3B-GGUF-3e4a3adb:/models:ro \
  --entrypoint /app/llama-server \
  ghcr.io/ggml-org/llama.cpp:server-cuda12-b11146@sha256:52192a23c7b258044463aebba78826cdff39a7393b713edc313324b343a7b836 \
  --model /models/Welp-35B-A3B.gguf \
  --fit-target 1024 \
  --flash-attn on \
  --ctx-size 262144 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --ubatch-size 256 \
  --parallel 1 \
  --cache-ram 0 \
  --metrics \
  --jinja \
  --host 0.0.0.0 \
  --port 8080

What it passed

✓ loadLoads and serves within an hour

✓ chatAnswers a plain question and stops on its own

✓ reasoningThinks separately and gets 17 × 23 right

✓ toolsCalls a tool with the right arguments and uses the result

✓ contextRecalls a code buried in a prompt that fills 85% of the window

✓ speedDecodes at 15 tok/s or more

Passed all six checks on a real RTX 5060 Ti on Oct 13, 2026.

Details

Weights: quaedra/Welp-35B-A3B-GGUF @ 3e4a3adbda ›
Image: ghcr.io/ggml-org/llama.cpp:server-cuda12-b11146@sha256:52192a23c7b258044463aebba78826cdff39a7393b713edc313324b343a7b836
Launch: llama.cpp-welp-35b-a3b-autofit-256k-rtx-5060-ti-16gb ›
Recipe file: registry/recipes/nvidia/rtx-5060-ti-16gb/welp-35b-a3b.llama.cpp.256k.json ›
Model released: Oct 8, 2026
Tested: owner, RTX 5060 Ti