Qwen3.8-Flash-Next

EXL3 3.05 bpw · sglang · 256K context · Arc Pro B70 · 32 GB

Qwen3.8's large mixture-of-experts model: far more knowledge than the 27B, with few active parameters, so it stays fast.

36 tok/sdecode

1,087 tok/sprefill

256Kcontext window

Tested on this cardOct 16, 2026

Run it

Follow the steps below to download and run this recipe. Omarchy Local AI lists the recipes it can launch on your hardware. The server then answers on http://localhost:8000/v1.

1 Set up first

# Requires one Intel Arc Pro B70 32 GB, 63 GB of free host RAM and 85.14 GB for the weights on a local NVMe SSD. Measured on omarchy (EPYC 7443P, 8-channel DDR4, PCIe 4.0 x16, md RAID0 of 4x Samsung 9100 PRO, xfs): system-wide MemAvailable drop 62.2 GB at peak (kit memwatch) - the routed experts (46.7 GB) live in USM host memory, which the xe driver does not charge to the container cgroup (cgroup peak ~11 GB), so the RAM need is the MemAvailable drop, not the container limit. The n-gram (engram) table (32.6 GB) is read from NVMe with O_DIRECT through a 1 GB RAM row cache; no expert NVMe tier (EXL3_NVTIER=0), outputs exact. NEXTN MTP 3 steps / 4 draft tokens (accept length ~2.1-2.5). Context 262,144 tokens with a 270,336-token fp8 KV cache. Vision (images). Kit protocol (local-ai-recipe-kit sweep): prefill 8k 1,141 / 16k 1,152 / 32k 1,145 / 64k 1,134 tok/s; decode C1 41.0, C2 42.3, C3 41.9, C4 41.7 tok/s aggregate, C1 at 32k 43.9; reference panel top-1 0.9883, KL 0.00117.
# Full instructions: https://github.com/sybil-solutions/qwen-serve/tree/d163a2f0dc47ab41c299eb8dca06520ef077b098/qwen3.8-flash-next/b70-offload

2 Download the weights

hf download turboderp/Qwen3.8-Flash-Next-exl3 \
  --revision 69e33439ae950f17bcbe95c98f117d80f759ab6d \
  --local-dir ~/models/Qwen3.8-Flash-Next-exl3-69e33439

3 Start the server

docker run --rm \
  --device /dev/dri \
  -p 8000:30250 \
  --memory 85899345920 \
  --memory-swap 85899345920 \
  --shm-size 64g \
  -e EXL3_LIB=/opt/trellis-serve/xpu/exl3xpu/_C_sgl.so \
  -e EXL3_MODEL_PATH=/models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5 \
  -e EXL3_MOE_CACHE=1 \
  -e EXL3_MOE_LIB=/opt/qwen-b70/kernels/moe-a8/_moe_a8.so \
  -e EXL3_MOE_SLOTS=8000 \
  -e EXL3_NGRAM_TIER=nvme \
  -e EXL3_NVTIER=0 \
  -e EXL3_QSA_DENSE_BYTES=134217728 \
  -e EXL3_QSA_ROWS=64 \
  -e EXL3_TORCH_MEM_FRACTION=0.95 \
  -e EXL3_VISION=1 \
  -e EnableSharedSystemUsmSupport=0 \
  -e HF_HUB_OFFLINE=1 \
  -e NEOReadDebugKeys=1 \
  -e QWEN_B70_MODE=nvme32 \
  -e SGLANG_EXL3_NGRAM_RAM_GB=1 \
  -e TORCHINDUCTOR_COMPILE_THREADS=1 \
  -e TreatNonUsmForTransfersAsSharedSystem=0 \
  -e ZE_AFFINITY_MASK=0 \
  -v ~/models/Qwen3.8-Flash-Next-exl3-69e33439:/models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5:ro \
  --entrypoint /opt/qwen-b70/serve/entrypoint.sh \
  ghcr.io/sybil-solutions/qwen38-flash-next-b70-offload@sha256:4f654928e73a77ff30e12cba5a366e092d02e88ca64b1c2d03d98353e8d19c5c \
  python3 \
  -m sglang.launch_server \
  --model-path /models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5 \
  --quantization exl3 \
  --trust-remote-code \
  --device xpu \
  --host 0.0.0.0 \
  --port 30250 \
  --served-model-name flashnext \
  --disable-shared-experts-fusion \
  --kv-cache-dtype fp8_e4m3 \
  --context-length 262144 \
  --mem-fraction-static 0.88 \
  --chunked-prefill-size 4096 \
  --max-running-requests 4 \
  --dtype bfloat16 \
  --max-mamba-cache-size 8 \
  --max-total-tokens 270336 \
  --cuda-graph-backend-decode full \
  --cuda-graph-bs-decode 1 \
  2 \
  4 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

What it passed

✓ loadLoads and serves within an hour

✓ chatAnswers a plain question and stops on its own

✓ reasoningThinks separately and gets 17 × 23 right

✓ toolsCalls a tool with the right arguments and uses the result

✓ contextRecalls a code buried in a prompt that fills 85% of the window

✓ speedDecodes at 15 tok/s or more

Passed all six checks on a real Intel Arc Pro B70 32GB on Oct 16, 2026.

Details

Weights: turboderp/Qwen3.8-Flash-Next-exl3 @ 69e33439ae ›
Image: ghcr.io/sybil-solutions/qwen38-flash-next-b70-offload@sha256:4f654928e73a77ff30e12cba5a366e092d02e88ca64b1c2d03d98353e8d19c5c
Source: sybil-solutions/qwen-serve @ d163a2f0dc47ab41c299eb8dca06520 ›
Launch: sglang-qwen3.8-flash-next-exl3-3.05bpw-mtp-ram80-256k-intel-arc-pro-b70-32gb ›
Recipe file: registry/recipes/intel/intel-arc-pro-b70-32gb/qwen3.8-flash-next.sglang.256k.json ›
Model released: Aug 24, 2026
Tested: owner, Intel Arc Pro B70 32GB