Welp-35B-A3B (GGUF)

A 13.21 GB GGUF of Qwen/Qwen3.6-35B-A3B that runs on a 16 GB GPU with the full 262k context, and scores higher than Unsloth's UD-IQ3_XXS at exactly the same size.

It uses only standard llama.cpp formats, the same ones as Unsloth's UD-IQ3_XXS, so it should run wherever that file runs. Tested with llama.cpp (PrismML build b10754 and a patched build of the same).

No fine-tuning or new training data: the same model, quantized more carefully.

File

Welp-35B-A3B.gguf, 13.21 GB. SHA-256 in SHA256SUMS.

This is the same per-tensor type mix as Unsloth's UD-IQ3_XXS. Only the expert weights are quantized differently (see below); everything else is byte-identical to Unsloth's file.

Running

Full 262k context on a 16 GB GPU (14.8 GB used, including about 0.8 GB for the desktop):

llama-server -m Welp-35B-A3B.gguf -ngl 99 -fa on -c 262144 -ctk q4_0 -ctv q4_0 -ub 256 --jinja

For a more precise KV cache at half the context: -c 131072 -ctk q8_0 -ctv q8_0.

Results

Models that fit entirely on a 16 GB GPU, measured on one RTX 4080 Super 16 GB with the same harness, single runs.

• HumanEval: 164 problems, thinking off, temperature 0, max 1024 tokens.
• Long-exact suite: bonsai-ada-surgery suite/ default plan, 37 long tool-using tasks, thinking on. Bonsai 2 scores 15/37 here against 17/37 in that repo's report (RTX 4070).
• Context: the largest that fits entirely on the card with a q4_0 KV cache.
• Decode: llama-bench tg128. Prefill: llama-bench pp4096.
• Perplexity is only comparable between Welp and UD-IQ3_XXS, which share the base model: wikitext-2 5.762 vs 5.873, code 1.880 vs 1.900 (context 2048, 40 chunks).
• Qwen3.8-27B is the dense model Bonsai 2 is built from. Welp solves one fewer HumanEval problem (157 vs 158) and three fewer long-exact tasks (23 vs 26), at 3.5 times its decode speed and twice its context. The HumanEval and suite differences against UD-IQ3_XXS are small enough to be run-to-run noise on their own; the perplexity gain is consistent.

HumanEval pass@1

Decode speed (tok/s, llama-bench tg128)

Prefill speed (tok/s, llama-bench pp4096)

HumanEval against GGUF size (log scale). The dashed line joins the best Qwen3.6-35B-A3B quantization at each size; Welp moves it up at 13.21 GB and ties Q4_K_M (157 of 164) at 62% of its size. The two ternary points are research builds from the Welp log and are not released.

At the full 262k context (q4_0 KV), after a 214k-token prompt: decode 107 tok/s, prefill 2,043 tok/s. A passcode hidden at 10%, 50% and 90% of that prompt was retrieved 3/3.

Speed against context

Decode tok/s (128 tokens) and prefill tok/s (next 2,048 tokens) after the given amount of context, llama-bench -d, q4_0 KV and -ub 256 for every model, RTX 4080 Super 16 GB, mean of 2 runs.

Decode speed against context

Prefill speed against context

• Welp keeps 65% of its decode speed at 252k; at 252k it still decodes faster than either 27B model with an empty cache. Only 10 of its 40 layers use full attention (the rest are Gated DeltaNet linear attention), and 3B of its parameters are active per token.
• -ub 256 is the batch size Welp needs for 262k on 16 GB, so prefill here is lower than the prefill column in the table above (pp4096, default batch). UD-IQ3_XXS has the same architecture and formats as Welp and runs at the same speed.

How it was made

Each expert matrix is quantized one 256-weight block at a time with llama.cpp's own quantizer, using that expert's activation statistics as the importance matrix. After each block, its rounding error is pushed into the columns not yet quantized (GPTQ, applied block-wise so any llama.cpp format works). Layers are processed in order, each fed the output of the already quantized layers. Calibration: 256 sequences of 2048 tokens, half code, half wikitext-2 train.

Recipe and scripts: github.com/quaedra/welp.

Credits

Base model by the Qwen team. The per-tensor type mix follows Unsloth's UD-IQ3_XXS, whose non-expert tensors this file reuses. Quantization formats from llama.cpp.

License

Apache 2.0, same as the base model.