Qwen3.8-Flash-Next

EXL3 3.05 bpw · sglang.256k.nvme12 · 256K context · Arc Pro B70 · 32 GB

Qwen3.8's large mixture-of-experts model: far more knowledge than the 27B, with few active parameters, so it stays fast.

30 tok/sdecode

873 tok/sprefill

256Kcontext window

Tested on this cardOct 17, 2026

Run it

Follow the steps below to download and run this recipe. Omarchy Local AI lists the recipes it can launch on your hardware. The server then answers on http://localhost:8000/v1.

1 Set up first

# Requires one Intel Arc Pro B70 32 GB, 12 GB of free host RAM (a 16 GB machine), and 132 GB on a local NVMe SSD: 85.14 GB of weights plus the 46.8 GB packed expert store that the image's pack-store step builds once (prepare; 49 layers including the MTP draft's MoE layer). Routed experts and the MTP layer's experts stay on NVMe behind the VRAM expert cache (8,000 slots) and a 0.5 GB RAM tier; a decode pick that is in neither is read from NVMe while the MoE kernel waits (exact decode miss path, EXL3_EXACT_MISS), so no expert pick is ever masked; VRAM evictions are not written back (EXL3_NVTIER_NO_VICTIM). One prefill staging buffer (EXL3_NVTIER_STAGE_HOSTBUF=1) and a 0.5 GB n-gram row cache keep the RAM down; the n-gram (engram) table is read from NVMe. NEXTN MTP 3 steps / 4 draft tokens. Context 262,144 tokens with a 270,336-token fp8 KV cache; vision (images). Measured on omarchy (EPYC 7443P, PCIe 4.0 x16, md RAID0 of 4x Samsung 9100 PRO, xfs): container-held RAM (anon + shmem PSS) peak 11.65 GB while serving (median 11.58); local-ai-recipe-kit sweep prefill 896-912 tok/s from 8K to 64K, decode 32.6-33.2 tok/s aggregate at 1-4 users, C1 at 32K 35.4; reference panel top-1 0.9883, KL 0.00118; needle 233,529 tokens pass. The container limit (16 GiB) leaves headroom for the shutdown path.
# Full instructions: https://github.com/sybil-solutions/qwen-serve/tree/730d8101d0ff7d53d7d6a3b980c3bae0c3b84181/qwen3.8-flash-next/b70-offload

2 Download the weights

hf download turboderp/Qwen3.8-Flash-Next-exl3 \
  --revision 69e33439ae950f17bcbe95c98f117d80f759ab6d \
  --local-dir ~/models/Qwen3.8-Flash-Next-exl3-69e33439

3 Prepare the model (first start)

mkdir -p ~/models/qwen3.8-flash-next.sglang.256k.nvme12-prepared-218f3fcbfc0a
docker run --rm \
  --user "$(id -u):$(id -g)" \
  -e HOME=/tmp \
  --memory 17179869184 \
  --memory-swap 17179869184 \
  --shm-size 64g \
  -e EXL3_EXACT_MISS=1 \
  -e EXL3_LIB=/opt/trellis-serve/xpu/exl3xpu/_C_sgl.so \
  -e EXL3_MODEL_PATH=/models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5 \
  -e EXL3_MOE_CACHE=1 \
  -e EXL3_MOE_LIB=/opt/qwen-b70/kernels/moe-a8/_moe_a8x.so \
  -e EXL3_MOE_SLOTS=8000 \
  -e EXL3_NGRAM_TIER=nvme \
  -e EXL3_NVTIER=1 \
  -e EXL3_NVTIER_NO_VICTIM=1 \
  -e EXL3_NVTIER_RAM_GB=0.5 \
  -e EXL3_NVTIER_STAGE_HOSTBUF=1 \
  -e EXL3_NVTIER_STAGE_M=16 \
  -e EXL3_NVTIER_STORE=/nvx/qwen_experts.bin \
  -e EXL3_QSA_DENSE_BYTES=134217728 \
  -e EXL3_QSA_ROWS=64 \
  -e EXL3_TORCH_MEM_FRACTION=0.95 \
  -e EXL3_VISION=1 \
  -e EnableRecoverablePageFaults=1 \
  -e EnableSharedSystemUsmSupport=1 \
  -e HF_HUB_OFFLINE=1 \
  -e MALLOC_ARENA_MAX=2 \
  -e NEOReadDebugKeys=1 \
  -e QWEN_B70_MODE=nvme16 \
  -e SGLANG_EXL3_NGRAM_RAM_GB=0.5 \
  -e TORCHINDUCTOR_COMPILE_THREADS=1 \
  -e TreatNonUsmForTransfersAsSharedSystem=0 \
  -e ZE_AFFINITY_MASK=0 \
  --entrypoint /opt/qwen-b70/serve/entrypoint.sh \
  -v ~/models/qwen3.8-flash-next.sglang.256k.nvme12-prepared-218f3fcbfc0a:/nvx \
  -v ~/models/Qwen3.8-Flash-Next-exl3-69e33439:/models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5:ro \
  ghcr.io/sybil-solutions/qwen38-flash-next-b70-offload@sha256:78bbcccc99cf09818b60ed08e7b2760e95d39802c2612b1a7ee0218f3fcbfc0a \
  pack-store

4 Verify the prepared model

docker run --rm --user "$(id -u):$(id -g)" -e HOME=/tmp --memory 17179869184 --memory-swap 17179869184 --shm-size 64g -e EXL3_EXACT_MISS=1 -e EXL3_LIB=/opt/trellis-serve/xpu/exl3xpu/_C_sgl.so -e EXL3_MODEL_PATH=/models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5 -e EXL3_MOE_CACHE=1 -e EXL3_MOE_LIB=/opt/qwen-b70/kernels/moe-a8/_moe_a8x.so -e EXL3_MOE_SLOTS=8000 -e EXL3_NGRAM_TIER=nvme -e EXL3_NVTIER=1 -e EXL3_NVTIER_NO_VICTIM=1 -e EXL3_NVTIER_RAM_GB=0.5 -e EXL3_NVTIER_STAGE_HOSTBUF=1 -e EXL3_NVTIER_STAGE_M=16 -e EXL3_NVTIER_STORE=/nvx/qwen_experts.bin -e EXL3_QSA_DENSE_BYTES=134217728 -e EXL3_QSA_ROWS=64 -e EXL3_TORCH_MEM_FRACTION=0.95 -e EXL3_VISION=1 -e EnableRecoverablePageFaults=1 -e EnableSharedSystemUsmSupport=1 -e HF_HUB_OFFLINE=1 -e MALLOC_ARENA_MAX=2 -e NEOReadDebugKeys=1 -e QWEN_B70_MODE=nvme16 -e SGLANG_EXL3_NGRAM_RAM_GB=0.5 -e TORCHINDUCTOR_COMPILE_THREADS=1 -e TreatNonUsmForTransfersAsSharedSystem=0 -e ZE_AFFINITY_MASK=0 --entrypoint /opt/qwen-b70/serve/entrypoint.sh -v ~/models/qwen3.8-flash-next.sglang.256k.nvme12-prepared-218f3fcbfc0a:/nvx:ro -v ~/models/Qwen3.8-Flash-Next-exl3-69e33439:/models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5:ro ghcr.io/sybil-solutions/qwen38-flash-next-b70-offload@sha256:78bbcccc99cf09818b60ed08e7b2760e95d39802c2612b1a7ee0218f3fcbfc0a verify-store

5 Start the server

docker run --rm \
  --device /dev/dri \
  -p 8000:30250 \
  --memory 17179869184 \
  --memory-swap 17179869184 \
  --shm-size 64g \
  -e EXL3_EXACT_MISS=1 \
  -e EXL3_LIB=/opt/trellis-serve/xpu/exl3xpu/_C_sgl.so \
  -e EXL3_MODEL_PATH=/models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5 \
  -e EXL3_MOE_CACHE=1 \
  -e EXL3_MOE_LIB=/opt/qwen-b70/kernels/moe-a8/_moe_a8x.so \
  -e EXL3_MOE_SLOTS=8000 \
  -e EXL3_NGRAM_TIER=nvme \
  -e EXL3_NVTIER=1 \
  -e EXL3_NVTIER_NO_VICTIM=1 \
  -e EXL3_NVTIER_RAM_GB=0.5 \
  -e EXL3_NVTIER_STAGE_HOSTBUF=1 \
  -e EXL3_NVTIER_STAGE_M=16 \
  -e EXL3_NVTIER_STORE=/nvx/qwen_experts.bin \
  -e EXL3_QSA_DENSE_BYTES=134217728 \
  -e EXL3_QSA_ROWS=64 \
  -e EXL3_TORCH_MEM_FRACTION=0.95 \
  -e EXL3_VISION=1 \
  -e EnableRecoverablePageFaults=1 \
  -e EnableSharedSystemUsmSupport=1 \
  -e HF_HUB_OFFLINE=1 \
  -e MALLOC_ARENA_MAX=2 \
  -e NEOReadDebugKeys=1 \
  -e QWEN_B70_MODE=nvme16 \
  -e SGLANG_EXL3_NGRAM_RAM_GB=0.5 \
  -e TORCHINDUCTOR_COMPILE_THREADS=1 \
  -e TreatNonUsmForTransfersAsSharedSystem=0 \
  -e ZE_AFFINITY_MASK=0 \
  -v ~/models/qwen3.8-flash-next.sglang.256k.nvme12-prepared-218f3fcbfc0a:/nvx:ro \
  -v ~/models/Qwen3.8-Flash-Next-exl3-69e33439:/models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5:ro \
  --entrypoint /opt/qwen-b70/serve/entrypoint.sh \
  ghcr.io/sybil-solutions/qwen38-flash-next-b70-offload@sha256:78bbcccc99cf09818b60ed08e7b2760e95d39802c2612b1a7ee0218f3fcbfc0a \
  python3 \
  -m sglang.launch_server \
  --model-path /models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5 \
  --quantization exl3 \
  --trust-remote-code \
  --device xpu \
  --host 0.0.0.0 \
  --port 30250 \
  --served-model-name flashnext \
  --disable-shared-experts-fusion \
  --kv-cache-dtype fp8_e4m3 \
  --context-length 262144 \
  --mem-fraction-static 0.88 \
  --chunked-prefill-size 4096 \
  --max-running-requests 4 \
  --dtype bfloat16 \
  --max-mamba-cache-size 8 \
  --max-total-tokens 270336 \
  --cuda-graph-backend-decode full \
  --cuda-graph-bs-decode 1 \
  2 \
  4 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

What it passed

✓ loadLoads and serves within an hour

✓ chatAnswers a plain question and stops on its own

✓ reasoningThinks separately and gets 17 × 23 right

✓ toolsCalls a tool with the right arguments and uses the result

✓ contextRecalls a code buried in a prompt that fills 85% of the window

✓ speedDecodes at 15 tok/s or more

Passed all six checks on a real Intel Arc Pro B70 32GB on Oct 17, 2026.

Details

Weights: turboderp/Qwen3.8-Flash-Next-exl3 @ 69e33439ae ›
Image: ghcr.io/sybil-solutions/qwen38-flash-next-b70-offload@sha256:78bbcccc99cf09818b60ed08e7b2760e95d39802c2612b1a7ee0218f3fcbfc0a
Source: sybil-solutions/qwen-serve @ 730d8101d0ff7d53d7d6a3b980c3bae ›
Launch: sglang-qwen3.8-flash-next-exl3-3.05bpw-mtp-nvme12-256k-intel-arc-pro-b70-32gb ›
Recipe file: registry/recipes/intel/intel-arc-pro-b70-32gb/qwen3.8-flash-next.sglang.256k.nvme12.json ›
Model released: Aug 24, 2026
Tested: owner, Intel Arc Pro B70 32GB