DeepSeek-V4-Flash

FP8 · vllm · 1024K context · 3 × DGX Spark GB10 · 384 GB

DeepSeek's V4 Flash mixture-of-experts model (and its 0731 and DSpark builds).

42 tok/sdecode

– tok/sprefill

1024Kcontext window

Reported by MiaAI-LabSep 16, 2026

Run it

This is the launch as MiaAI-Lab publishes it. Their repository has the full setup: MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark@97e8733238f8 › The server then answers on http://localhost:8000/v1.

1 Set up first

# ./start-tp3.sh (sets DSPARK_TP3=1 TP_SIZE=3 NNODES=3); needs WORKER2_HOST etc. in .env.dspark; three Sparks in a QSFP ring, both CX ports used (NCCL_IB_HCA=rocep1s0f0,rocep1s0f1), bootstrap Gloo/TP sockets over the shared RJ45 LAN iface enP7s7; TP3 group-pad patch patches/tp3/apply_tp3_patch.py applied at container start; ranks 1 and 2 headless
# Full instructions: https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/tree/97e8733238f81f5fdc44b241f8996a7858825744

2 Download the weights

hf download deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
  --revision 86f746b36186f0e567729a5c06a8c918caba82a9

3 Start the server on each of the 3 machines (NODE_RANK 0 to 2)

docker run --rm \
  --gpus all \
  -p 8000:8888 \
  --network host \
  --ipc host \
  --gpus all \
  --device /dev/infiniband \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --shm-size 64gb \
  -e B12X_CUTE_COMPILE_CACHE_DIR=/cache/huggingface/b12x-cute-cache \
  -e CUTE_DSL_ARCH=sm_121a \
  -e DEFAULT_THINKING=low \
  -e DG_JIT_NVCC_COMPILER=/usr/local/cuda/bin/nvcc \
  -e DG_JIT_USE_NVRTC=0 \
  -e DRAFT_SAMPLE_METHOD=probabilistic \
  -e DSPARK_ASYNC_SCHEDULING=1 \
  -e DSPARK_MAX_INFLIGHT_PREFILLS=1 \
  -e DSPARK_MODEL=deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
  -e DSPARK_REVISION=86f746b36186f0e567729a5c06a8c918caba82a9 \
  -e DSPARK_SUPPRESS_STOPS_IN_REASONING=1 \
  -e FLASHINFER_CUDA_ARCH_LIST=12.1a \
  -e FLASHINFER_DISABLE_VERSION_CHECK=1 \
  -e FLASHINFER_WORKSPACE_BASE=/cache/huggingface/flashinfer \
  -e GLOO_SOCKET_IFNAME=enP7s7 \
  -e GPU_MEMORY_UTILIZATION=0.835 \
  -e HEADLESS="${HEADLESS}" \
  -e HF_HOME=/cache/huggingface \
  -e HF_HUB_DISABLE_XET=1 \
  -e HF_HUB_OFFLINE=1 \
  -e MASTER_ADDR="${MASTER_ADDR}" \
  -e MASTER_PORT=25000 \
  -e MTP_NUM_TOKENS=6 \
  -e NCCL_CROSS_NIC=1 \
  -e NCCL_CUMEM_ENABLE=0 \
  -e NCCL_DEBUG=WARN \
  -e NCCL_IB_ADDR_FAMILY=AF_INET \
  -e NCCL_IB_DISABLE=0 \
  -e NCCL_IB_GID_INDEX="${NCCL_IB_GID_INDEX}" \
  -e NCCL_IB_HCA=rocep1s0f0,rocep1s0f1 \
  -e NCCL_IB_MERGE_NICS=0 \
  -e NCCL_IB_ROCE_VERSION_NUM=2 \
  -e NCCL_IB_SUBNET_AWARE_ROUTING=1 \
  -e NCCL_IB_SUBNET_PREFIX_LEN=24 \
  -e NCCL_IGNORE_CPU_AFFINITY=1 \
  -e NCCL_NET=IB \
  -e NCCL_NVLS_ENABLE=0 \
  -e NCCL_SOCKET_IFNAME=enP7s7 \
  -e NNODES=3 \
  -e NODE_RANK="${NODE_RANK}" \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e TILELANG_CACHE_DIR=/cache/huggingface/tilelang-cache \
  -e TILELANG_CLEANUP_TEMP_FILES=1 \
  -e TORCH_CUDA_ARCH_LIST=12.1a \
  -e TORCH_FR_BUFFER_SIZE=2000 \
  -e TORCH_NCCL_DUMP_ON_TIMEOUT=1 \
  -e TORCH_NCCL_ENABLE_MONITORING=1 \
  -e TP_SIZE=3 \
  -e TP_SOCKET_IFNAME=enP7s7 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e TRITON_CACHE_DIR=/cache/huggingface/triton-cache \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e VLLM_B12X_W4A16_FORCE_BLOCKS_MAX_M=16 \
  -e VLLM_B12X_W4A16_FORCE_BLOCKS_PER_SM=0 \
  -e VLLM_CACHE_ROOT=/cache/huggingface/vllm-cache \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_HOST_IP="${VLLM_HOST_IP}" \
  -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \
  -e VLLM_PREFIX_CACHE_RETENTION_INTERVAL=4096 \
  -e VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=256 \
  -e VLLM_USE_B12X_MOE=1 \
  -e VLLM_USE_BREAKABLE_CUDAGRAPH=0 \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint /usr/local/bin/vllm \
  ghcr.io/anemll/dspark-vllm-gx10:0.1.1@sha256:a83948492cf13df455170fb42885f5ef4db54fefe0feff0f841ecbff464ac9d8 \
  serve \
  deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
  --revision 86f746b36186f0e567729a5c06a8c918caba82a9 \
  --served-model-name deepseek-v4-flash-vision-exp \
  --host 0.0.0.0 \
  --port 8888 \
  --trust-remote-code \
  --tensor-parallel-size 3 \
  --pipeline-parallel-size 1 \
  --kv-cache-dtype nvfp4_ds_mla \
  --block-size 256 \
  --max-model-len 1048576 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --long-prefill-token-threshold 1024 \
  --max-cudagraph-capture-size 112 \
  --gpu-memory-utilization 0.835 \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --async-scheduling \
  --enable-chunked-prefill \
  --speculative-config '{"method":"dspark","num_speculative_tokens":6,"draft_sample_method":"probabilistic"}' \
  --tokenizer-mode deepseek_v4 \
  --limit-mm-per-prompt '{"image":8}' \
  --distributed-executor-backend mp \
  --moe-backend flashinfer_b12x \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"<think>","reasoning_end_str":"</think>"}' \
  --default-chat-template-kwargs '{"thinking":true,"reasoning_effort":"low"}' \
  --generation-config vllm \
  --enable-flashinfer-autotune \
  --nnodes 3 \
  --node-rank "${NODE_RANK}" \
  --master-addr "${MASTER_ADDR}" \
  --master-port 25000 \
  '${HEADLESS:+--headless}'

What it passed

– loadLoads and serves within an hour

– chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by MiaAI-Lab in MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark@97e8733238f8, where it reports 41.9 tok/s. Our six checks have not run on it yet.

Details

Weights: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp @ 86f746b361 ›
Image: ghcr.io/anemll/dspark-vllm-gx10:0.1.1@sha256:a83948492cf13df455170fb42885f5ef4db54fefe0feff0f841ecbff464ac9d8
Source: MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark @ 97e8733238 ›
Launch: vllm-deepseek-v4-flash-fp8-1024k-3x ›
Recipe file: registry/recipes/nvidia/dgx-spark-gb10-128gb/deepseek-v4-flash.vllm.1024k.3x.json ›
Model released: Apr 22, 2026
Tested: not yet by us; reported by miaai-lab