DeepSeek-V4-Flash

EXL3 3 bpw · vllm · 375K context · DGX Spark GB10 · 128 GB

DeepSeek's V4 Flash mixture-of-experts model (and its 0731 and DSpark builds).

46 tok/sdecode

625 tok/sprefill

375Kcontext window

Reported by MiaAI-LabAug 24, 2026

Run it

This is the launch as MiaAI-Lab publishes it. Their repository has the full setup: MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark@fdcd538fbf95 › The server then answers on http://localhost:8000/v1.

1 Set up first

# start.sh generates compose.yml (not shipped). First boot downloads ~107 GB into ./hf-hub (bound at /hf-cache), coalesces rank-sliced TP4 EXL3 -> TP1 into ./data/tp1 (/models), builds K64 DSpark draft, runs selftest. Optional ./download.sh does download+coalesce offline. Bind-mounts from image-patch/: entrypoint-256k.sh -> /opt/recipe/scripts/entrypoint.sh, entrypoint-toolfix.sh -> /patch-run-entrypoint.sh (pip upgrades xgrammar>=0.2.4, pins transformers==5.13.1 at boot), serve-ds4-flash.sh -> /opt/vllm/serve-ds4-flash.sh, coalesce_rank_sliced_exl3.py, dspark.py, and sparkinfer kernel backports (_impl.py, tiny_decode.py, prefill.py, prefill_mg.py). ./cache -> /cache. Disable earlyoom; needs >=114.3 GiB free host RAM at launch; aarch64 only.
# Full instructions: https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark/tree/fdcd538fbf95fb15b2d6850db9613d22b2c889b8

2 Download the weights

hf download 0xSero/deepseek-v4-flash-0731-spark \
  --revision 22f28d32b9b29b4352eaa380ff8c2c170b2847ab

3 Start the server

docker run --rm \
  --gpus all \
  -p 8000:8888 \
  network_mode: host \
  ipc: host \
  restart: on-failure:1 \
  pull_policy: always \
  deploy.resources.reservations.devices nvidia count 1 \
  --shm-size 16gb \
  -e CUDAGRAPH_CAPTURE_SIZES=6,12,24 \
  -e DEFAULT_CHAT_TEMPLATE_KWARGS_EFFORT=max \
  -e DEFAULT_CHAT_TEMPLATE_KWARGS_THINKING=true \
  -e DSPARK_CAPACITY=0 \
  -e DSPARK_DRAFT_EXPERTS=64 \
  -e DSPARK_DYNAMIC_DRAFT_DEPTH=0 \
  -e DSPARK_DYNAMIC_DRAFT_DEPTH_WINDOW=8 \
  -e DSPARK_STRUCTURED_EXPERTS_PER_CATEGORY=32 \
  -e DSPARK_TOKENS=5 \
  -e EXTRA_VLLM_ARGS='--long-prefill-token-threshold 1024' \
  -e GPU_MEMORY_UTILIZATION=0.94 \
  -e HF_HOME=/hf-cache \
  -e HF_TOKEN='${HF_TOKEN:-}' \
  -e HOST=0.0.0.0 \
  -e KV_FP8_ROPE=0 \
  -e KV_OFFLOAD_GB=0 \
  -e MAX_CUDAGRAPH_CAPTURE_SIZE=24 \
  -e MAX_MODEL_LEN=384000 \
  -e MAX_NUM_BATCHED_TOKENS=8224 \
  -e MAX_NUM_SEQS=1 \
  -e MODE=dspark \
  -e MODEL_REPO=0xSero/deepseek-v4-flash-0731-spark \
  -e MODEL_REVISION=22f28d32b9b29b4352eaa380ff8c2c170b2847ab \
  -e MODEL_SOURCE_DIR=/hf-cache/hub/models--0xSero--deepseek-v4-flash-0731-spark/snapshots/22f28d32b9b29b4352eaa380ff8c2c170b2847ab \
  -e PORT=8888 \
  -e SERVED_MODEL_NAME=deepseek-v4-flash-0731 \
  -e VERIFY_MODEL_CHECKSUMS=1 \
  -e VLLM_DSV4_PADDED_NVFP4=0 \
  -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \
  -e VLLM_USE_B12X_WO_PROJECTION=1 \
  -e VLLM_USE_SIMPLE_KV_OFFLOAD=0 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint /bin/bash \
  ghcr.io/0xsero/deepseek-v4-flash-0731-spark-sparkinfer@sha256:2e077489a83a0360952828051fe7f7a32c1801e5ce8436d85f7267583d614ff4 \
  -lc 'bash /patch-run-entrypoint.sh && exec /opt/recipe/scripts/entrypoint.sh' \
  serve \
  /models/tp1 \
  --served-model-name deepseek-v4-flash-0731 \
  --host 0.0.0.0 \
  --port 8888 \
  --trust-remote-code \
  --kv-cache-dtype nvfp4_ds_mla \
  --block-size 256 \
  --load-format instanttensor \
  --tensor-parallel-size 1 \
  --decode-context-parallel-size 1 \
  --gpu-memory-utilization 0.94 \
  --max-model-len 384000 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 8224 \
  --max-cudagraph-capture-size 24 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --async-scheduling \
  --no-scheduler-reserve-full-isl \
  --enable-chunked-prefill \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --enable-prompt-tokens-details \
  --enable-force-include-usage \
  --enable-request-id-headers \
  --default-chat-template-kwargs.thinking=true \
  --default-chat-template-kwargs.reasoning_effort=max \
  --enable-flashinfer-autotune \
  --enable-prefix-caching \
  --cudagraph-capture-sizes 6 \
  12 \
  24 \
  --speculative-config '{"model":"/models/dspark-draft-k64","method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"standard","attention_backend":"B12X_MLA_SPARSE"}' \
  --attention-backend B12X_MLA_SPARSE \
  --moe-backend b12x \
  --linear-backend b12x \
  --disable-custom-all-reduce \
  --long-prefill-token-threshold 1024

What it passed

– loadLoads and serves within an hour

– chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by MiaAI-Lab in MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark@fdcd538fbf95, where it reports 45.5 tok/s. Our six checks have not run on it yet.

Details

Weights: 0xSero/deepseek-v4-flash-0731-spark @ 22f28d32b9 ›
Image: ghcr.io/0xsero/deepseek-v4-flash-0731-spark-sparkinfer@sha256:2e077489a83a0360952828051fe7f7a32c1801e5ce8436d85f7267583d614ff4
Source: MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark @ fdcd538fbf95fb15 ›
Launch: vllm-deepseek-v4-flash-exl3-3.0bpw-375k ›
Recipe file: registry/recipes/nvidia/dgx-spark-gb10-128gb/deepseek-v4-flash.vllm.375k.json ›
Model released: Apr 22, 2026
Tested: not yet by us; reported by miaai-lab