Qwen3.8-27B-AWQ-BF16-INT4

· vllm · 256K context · DGX Spark GB10 · 128 GB

cyankiwi's AWQ INT4 quant of Qwen3.8-27B, linear attention kept in BF16.

35 tok/sdecode

582 tok/sprefill

256Kcontext window

Reported by toncaoSep 30, 2026

Run it

This is the launch as toncao publishes it. Their repository has the full setup: toncao/local-ai-registry@f5be8bf9c62b449a7fe1c82ce01f9bd10426e861 › The server then answers on http://localhost:8000/v1.

1 Download the weights

hf download cyankiwi/Qwen3.8-27B-AWQ-BF16-INT4 \
  --revision dc430725f831dd90d9271738b877879a46a82239 \
  --local-dir ~/models/Qwen3.8-27B-AWQ-BF16-INT4-dc430725

2 Start the server

docker run --rm \
  --gpus all \
  -p 8000:8000 \
  --shm-size 16g \
  -v ~/models/Qwen3.8-27B-AWQ-BF16-INT4-dc430725:/models/Qwen3.8-27B-AWQ-BF16-INT4-dc430725:ro \
  --entrypoint vllm \
  vllm/vllm-openai:v0.30.0@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90 \
  serve \
  /models/Qwen3.8-27B-AWQ-BF16-INT4-dc430725 \
  --served-model-name Qwen3.8-27B-AWQ-BF16-INT4-dc430725 \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.8 \
  --kv-cache-dtype fp8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","revision":"015e795645c74b1a0eeef3b570031fb62e769bc5","num_speculative_tokens":7}' \
  --safetensors-load-strategy eager

What it passed

– loadLoads and serves within an hour

– chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by toncao in toncao/local-ai-registry@f5be8bf9c62b449a7fe1c82ce01f9bd10426e861, where it reports 34.5 tok/s. Our six checks have not run on it yet. Reported decode averages the entire answer; requests omitted temperature. Canonical lab acceptance is pending.

Details

Weights: cyankiwi/Qwen3.8-27B-AWQ-BF16-INT4 @ dc430725f8 ›
Image: vllm/vllm-openai:v0.30.0@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90
Source: toncao/local-ai-registry @ f5be8bf9c62b449a7fe1c82ce01f9bd10 ›
Engine profile: vllm ›
Recipe file: registry/recipes/nvidia/dgx-spark-gb10-128gb/qwen3.8-27b-awq-bf16-int4.vllm.256k.json ›
Model released: Aug 17, 2026
Tested: not yet by us; reported by toncao