Qwen3.8-Flash-Next

EXL3 3.05 bpw · sglang.256k.nvme58 · 256K context · Arc Pro B70 · 32 GB

Qwen3.8's large mixture-of-experts model: far more knowledge than the 27B, with few active parameters, so it stays fast.

32 tok/sdecode

1,020 tok/sprefill

256Kcontext window

Tested on this cardOct 17, 2026

Run it

Follow the steps below to download and run this recipe. Omarchy Local AI lists the recipes it can launch on your hardware. The server then answers on http://localhost:8000/v1.

1 Set up first

# Requires one Intel Arc Pro B70 32 GB, 54 GB of free host RAM, and 131 GB on a local NVMe SSD: 85.14 GB of weights plus the 45.8 GB packed expert store that the image's pack-store step builds once (prepare). Routed experts stay on NVMe behind the VRAM expert cache (8,000 slots) and a 36 GB RAM tier; a decode pick that is in neither is read from NVMe while the MoE kernel waits (exact decode miss path, EXL3_EXACT_MISS), so no expert pick is ever masked; VRAM evictions are not written back (EXL3_NVTIER_NO_VICTIM). The n-gram (engram) table is read from NVMe. NEXTN MTP 3 steps / 4 draft tokens (the MTP layer's 512 experts stay pinned in the RAM tier). Context 262,144 tokens with a 270,336-token fp8 KV cache; vision (images). Measured on omarchy (EPYC 7443P, PCIe 4.0 x16, md RAID0 of 4x Samsung 9100 PRO, xfs): container-held RAM (anon + shmem PSS) peak 51.7 GB while serving; local-ai-recipe-kit sweep prefill 982-1,076 tok/s from 8K to 64K, decode 36.9-38.0 tok/s aggregate at 1-4 users, C1 at 32K 39.0; reference panel top-1 0.9877, KL 0.00125; needle 233,529 tokens pass. The container limit leaves headroom for the shutdown path.
# Full instructions: https://github.com/sybil-solutions/qwen-serve/tree/b257e3e51a77ef052e18fafab2687d5f8f873931/qwen3.8-flash-next/b70-offload

2 Download the weights

hf download turboderp/Qwen3.8-Flash-Next-exl3 \
  --revision 69e33439ae950f17bcbe95c98f117d80f759ab6d \
  --local-dir ~/models/Qwen3.8-Flash-Next-exl3-69e33439

3 Prepare the model (first start)

mkdir -p ~/models/qwen3.8-flash-next.sglang.256k.nvme58-prepared-d7d13662c3a5
docker run --rm \
  --user "$(id -u):$(id -g)" \
  -e HOME=/tmp \
  --memory 70866960384 \
  --memory-swap 70866960384 \
  --shm-size 64g \
  -e EXL3_EXACT_MISS=1 \
  -e EXL3_LIB=/opt/trellis-serve/xpu/exl3xpu/_C_sgl.so \
  -e EXL3_MODEL_PATH=/models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5 \
  -e EXL3_MOE_CACHE=1 \
  -e EXL3_MOE_LIB=/opt/qwen-b70/kernels/moe-a8/_moe_a8x.so \
  -e EXL3_MOE_SLOTS=8000 \
  -e EXL3_NGRAM_TIER=nvme \
  -e EXL3_NVTIER=1 \
  -e EXL3_NVTIER_NO_VICTIM=1 \
  -e EXL3_NVTIER_RAM_GB=36 \
  -e EXL3_NVTIER_STAGE_M=16 \
  -e EXL3_NVTIER_STORE=/nvx/qwen_experts.bin \
  -e EXL3_QSA_DENSE_BYTES=134217728 \
  -e EXL3_QSA_ROWS=64 \
  -e EXL3_TORCH_MEM_FRACTION=0.95 \
  -e EXL3_VISION=1 \
  -e EnableRecoverablePageFaults=1 \
  -e EnableSharedSystemUsmSupport=1 \
  -e HF_HUB_OFFLINE=1 \
  -e MALLOC_ARENA_MAX=2 \
  -e NEOReadDebugKeys=1 \
  -e QWEN_B70_MODE=nvme32 \
  -e TORCHINDUCTOR_COMPILE_THREADS=1 \
  -e TreatNonUsmForTransfersAsSharedSystem=0 \
  -e ZE_AFFINITY_MASK=0 \
  --entrypoint /opt/qwen-b70/serve/entrypoint.sh \
  -v ~/models/qwen3.8-flash-next.sglang.256k.nvme58-prepared-d7d13662c3a5:/nvx \
  -v ~/models/Qwen3.8-Flash-Next-exl3-69e33439:/models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5:ro \
  ghcr.io/sybil-solutions/qwen38-flash-next-b70-offload@sha256:e9104c18e2b6f639db4612d0327d96ba3a6ecf2d4a7aa8736823d7d13662c3a5 \
  pack-store

4 Verify the prepared model

docker run --rm --user "$(id -u):$(id -g)" -e HOME=/tmp --memory 70866960384 --memory-swap 70866960384 --shm-size 64g -e EXL3_EXACT_MISS=1 -e EXL3_LIB=/opt/trellis-serve/xpu/exl3xpu/_C_sgl.so -e EXL3_MODEL_PATH=/models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5 -e EXL3_MOE_CACHE=1 -e EXL3_MOE_LIB=/opt/qwen-b70/kernels/moe-a8/_moe_a8x.so -e EXL3_MOE_SLOTS=8000 -e EXL3_NGRAM_TIER=nvme -e EXL3_NVTIER=1 -e EXL3_NVTIER_NO_VICTIM=1 -e EXL3_NVTIER_RAM_GB=36 -e EXL3_NVTIER_STAGE_M=16 -e EXL3_NVTIER_STORE=/nvx/qwen_experts.bin -e EXL3_QSA_DENSE_BYTES=134217728 -e EXL3_QSA_ROWS=64 -e EXL3_TORCH_MEM_FRACTION=0.95 -e EXL3_VISION=1 -e EnableRecoverablePageFaults=1 -e EnableSharedSystemUsmSupport=1 -e HF_HUB_OFFLINE=1 -e MALLOC_ARENA_MAX=2 -e NEOReadDebugKeys=1 -e QWEN_B70_MODE=nvme32 -e TORCHINDUCTOR_COMPILE_THREADS=1 -e TreatNonUsmForTransfersAsSharedSystem=0 -e ZE_AFFINITY_MASK=0 --entrypoint /opt/qwen-b70/serve/entrypoint.sh -v ~/models/qwen3.8-flash-next.sglang.256k.nvme58-prepared-d7d13662c3a5:/nvx:ro -v ~/models/Qwen3.8-Flash-Next-exl3-69e33439:/models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5:ro ghcr.io/sybil-solutions/qwen38-flash-next-b70-offload@sha256:e9104c18e2b6f639db4612d0327d96ba3a6ecf2d4a7aa8736823d7d13662c3a5 verify-store

5 Start the server

docker run --rm \
  --device /dev/dri \
  -p 8000:30250 \
  --memory 70866960384 \
  --memory-swap 70866960384 \
  --shm-size 64g \
  -e EXL3_EXACT_MISS=1 \
  -e EXL3_LIB=/opt/trellis-serve/xpu/exl3xpu/_C_sgl.so \
  -e EXL3_MODEL_PATH=/models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5 \
  -e EXL3_MOE_CACHE=1 \
  -e EXL3_MOE_LIB=/opt/qwen-b70/kernels/moe-a8/_moe_a8x.so \
  -e EXL3_MOE_SLOTS=8000 \
  -e EXL3_NGRAM_TIER=nvme \
  -e EXL3_NVTIER=1 \
  -e EXL3_NVTIER_NO_VICTIM=1 \
  -e EXL3_NVTIER_RAM_GB=36 \
  -e EXL3_NVTIER_STAGE_M=16 \
  -e EXL3_NVTIER_STORE=/nvx/qwen_experts.bin \
  -e EXL3_QSA_DENSE_BYTES=134217728 \
  -e EXL3_QSA_ROWS=64 \
  -e EXL3_TORCH_MEM_FRACTION=0.95 \
  -e EXL3_VISION=1 \
  -e EnableRecoverablePageFaults=1 \
  -e EnableSharedSystemUsmSupport=1 \
  -e HF_HUB_OFFLINE=1 \
  -e MALLOC_ARENA_MAX=2 \
  -e NEOReadDebugKeys=1 \
  -e QWEN_B70_MODE=nvme32 \
  -e TORCHINDUCTOR_COMPILE_THREADS=1 \
  -e TreatNonUsmForTransfersAsSharedSystem=0 \
  -e ZE_AFFINITY_MASK=0 \
  -v ~/models/qwen3.8-flash-next.sglang.256k.nvme58-prepared-d7d13662c3a5:/nvx:ro \
  -v ~/models/Qwen3.8-Flash-Next-exl3-69e33439:/models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5:ro \
  --entrypoint /opt/qwen-b70/serve/entrypoint.sh \
  ghcr.io/sybil-solutions/qwen38-flash-next-b70-offload@sha256:e9104c18e2b6f639db4612d0327d96ba3a6ecf2d4a7aa8736823d7d13662c3a5 \
  python3 \
  -m sglang.launch_server \
  --model-path /models/turboderp-Qwen3.8-Flash-Next-exl3-3.05bpw_h5_ng5 \
  --quantization exl3 \
  --trust-remote-code \
  --device xpu \
  --host 0.0.0.0 \
  --port 30250 \
  --served-model-name flashnext \
  --disable-shared-experts-fusion \
  --kv-cache-dtype fp8_e4m3 \
  --context-length 262144 \
  --mem-fraction-static 0.88 \
  --chunked-prefill-size 4096 \
  --max-running-requests 4 \
  --dtype bfloat16 \
  --max-mamba-cache-size 8 \
  --max-total-tokens 270336 \
  --cuda-graph-backend-decode full \
  --cuda-graph-bs-decode 1 \
  2 \
  4 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

What it passed

✓ loadLoads and serves within an hour

✓ chatAnswers a plain question and stops on its own

✓ reasoningThinks separately and gets 17 × 23 right

✓ toolsCalls a tool with the right arguments and uses the result

✓ contextRecalls a code buried in a prompt that fills 85% of the window

✓ speedDecodes at 15 tok/s or more

Passed all six checks on a real Intel Arc Pro B70 32GB on Oct 17, 2026.

Details

Weights: turboderp/Qwen3.8-Flash-Next-exl3 @ 69e33439ae ›
Image: ghcr.io/sybil-solutions/qwen38-flash-next-b70-offload@sha256:e9104c18e2b6f639db4612d0327d96ba3a6ecf2d4a7aa8736823d7d13662c3a5
Source: sybil-solutions/qwen-serve @ b257e3e51a77ef052e18fafab2687d5 ›
Launch: sglang-qwen3.8-flash-next-exl3-3.05bpw-mtp-nvme58-256k-intel-arc-pro-b70-32gb ›
Recipe file: registry/recipes/intel/intel-arc-pro-b70-32gb/qwen3.8-flash-next.sglang.256k.nvme58.json ›
Model released: Aug 24, 2026
Tested: owner, Intel Arc Pro B70 32GB