DeepSeek-V4.1-Flash

· vllm · 256K context · 2 × DGX Spark GB10 · 256 GB

DeepSeek's V4.1 Flash, a large mixture-of-experts model.

23 tok/sdecode

1,939 tok/sprefill

256Kcontext window

Tested on this cardOct 3, 2026

Run it

Follow the steps below to download and run this recipe. Omarchy Local AI lists the recipes it can launch on your hardware. The server then answers on http://localhost:8000/v1.

1 Set up first

# Head (rank 0, API) + worker (rank 1, --headless, same args with NODE_RANK=1) containers over the CX7 RoCE fabric (--nnodes 2, mp executor). Set NODE_RANK, HEAD_IP, NODE_IP, CX7_IF and CX7_IB on each node; NODE_IP must be its reachable fabric IP. Both nodes download the pinned 0xSero/DeepSeek-V4.1-Flash-Spark repository to local NVMe and mount it at /model. This public repository includes the native checkpoint, Engram parts and EXL3 banks; no separate full upstream checkpoint is needed. After download, run bash ~/models/DeepSeek-V4.1-Flash-Spark-08ac8b3d/reassemble.sh once on each node to reconstruct and sha256-check the two Engram shards. Allow about 665 GB for download plus reassembly, in addition to image/cache space; after a successful check the .partNN files may be deleted, leaving about 462 GB. The plan is /model/exl3/plan/J268-ho-v31-K5n.json; its qdir is qn:q31:q31s relative to --workdir /model/exl3. Mount a persistent /cache for b12x and RoCE JIT caches. The source repository includes a host memory guard to stop containers before a GB10 unified-memory OOM. Load about 12 minutes. CANDIDATE: the earlier S018 lab run used the original local full-checkpoint layout and the unattested host image; the final published layout and approved CI image still require a fresh six-gate run.
# Full instructions: https://github.com/0xSero/local-ai-images/tree/main/deepseek-v4.1-flash-spark

2 Download the weights

hf download 0xSero/DeepSeek-V4.1-Flash-Spark \
  --revision 08ac8b3defc9a239ba0baf51059687c689406519 \
  --local-dir ~/models/DeepSeek-V4.1-Flash-Spark-08ac8b3d

3 Start the server on each of the 2 machines (NODE_RANK 0 to 1)

docker run --rm \
  --gpus all \
  -p 8000:8000 \
  --gpus all \
  --network host \
  --ipc=host \
  --privileged \
  --shm-size 32g \
  --ulimit memlock=-1 \
  --device /dev/infiniband \
  --workdir /model/exl3 \
  --shm-size 32g \
  -e B12X_COMPILE_CACHE_DIR=/cache/b12x \
  -e B12X_ROCE_CACHE_DIR=/cache/b12x-roce \
  -e B12X_ROCE_TRAFFIC_CLASS=106 \
  -e CUDA_VISIBLE_DEVICES=0 \
  -e CUTE_DSL_ARCH=sm_121a \
  -e GLOO_SOCKET_IFNAME="${CX7_IF}" \
  -e HF_HUB_OFFLINE=1 \
  -e MALLOC_ARENA_MAX=2 \
  -e NCCL_DEBUG=WARN \
  -e NCCL_IB_DISABLE=0 \
  -e NCCL_IB_GID_INDEX=3 \
  -e NCCL_IB_HCA="${CX7_IB}" \
  -e NCCL_IB_MERGE_NICS=1 \
  -e NCCL_IB_TC=106 \
  -e NCCL_NET_PLUGIN=none \
  -e NCCL_SOCKET_IFNAME="${CX7_IF}" \
  -e OMP_NUM_THREADS=16 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e SAFETENSORS_FAST_GPU=1 \
  -e ST_EXL3_PLAN=/model/exl3/plan/J268-ho-v31-K5n.json \
  -e ST_EXL3_PREFILL=st \
  -e ST_EXL3_PREFILL_MIN_ROWS=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e VLLM_ENABLE_PCIE_ALLREDUCE=0 \
  -e VLLM_ENABLE_ROCE_ALLREDUCE=1 \
  -e VLLM_HOST_IP="${NODE_IP}" \
  -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 \
  -e VLLM_ROCE_ALLGATHER_MAX_SIZE=16MB \
  -e VLLM_ROCE_ALLREDUCE_MAX_SIZE=2MB \
  -e VLLM_USE_BREAKABLE_CUDAGRAPH=0 \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e XDG_CACHE_HOME=/cache \
  -v ~/models/DeepSeek-V4.1-Flash-Spark-08ac8b3d:/model:ro \
  --entrypoint vllm \
  ghcr.io/0xsero/deepseek-v4.1-flash-spark@sha256:3cbc8ec016f5fbfc82eba3480de12399cbce31e7b76aa3108b8fe8246c42a79d \
  serve \
  /model \
  --served-model-name deepseek-v4.1-flash \
  --dtype bfloat16 \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --nnodes 2 \
  --node-rank "${NODE_RANK}" \
  --master-addr "${HEAD_IP}" \
  --master-port 29655 \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --swa-block-size 128 \
  --kv-cache-memory-bytes 2700000000 \
  --gpu-memory-utilization 0.88 \
  --max-model-len 262144 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 2048 \
  --max-cudagraph-capture-size 16 \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","custom_ops":["all"],"cudagraph_capture_sizes":[1,2,3,4,5,6,7,8,12,16]}' \
  --enable-chunked-prefill \
  --async-scheduling \
  --enable-prefix-caching \
  --safetensors-load-strategy lazy \
  --engram-config '{"cpu_offload":false,"table_memory":"disk","disk_resident_scales":false,"disk_prefetch_max_tokens":0,"projection_tp":false}' \
  --attention-backend B12X \
  --linear-backend b12x \
  --moe-backend b12x \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_tensor_parallel_size":2,"attention_backend":"B12X","draft_sample_method":"greedy","rejection_sample_method":"standard","enable_adaptive_verification":true}' \
  --generation-config vllm \
  --tokenizer-mode deepseek_v41 \
  --limit-mm-per-prompt '{"image":1}' \
  --reasoning-parser deepseek_v41 \
  --tool-call-parser deepseek_v41 \
  --enable-auto-tool-choice \
  --enable-prompt-tokens-details \
  --enable-force-include-usage

What it passed

✓ loadLoads and serves within an hour

✓ chatAnswers a plain question and stops on its own

✓ reasoningThinks separately and gets 17 × 23 right

✓ toolsCalls a tool with the right arguments and uses the result

✓ contextRecalls a code buried in a prompt that fills 85% of the window

✓ speedDecodes at 15 tok/s or more

Passed all six checks on a real 2x DGX Spark (GB10) on Oct 3, 2026.

Details

Weights: 0xSero/DeepSeek-V4.1-Flash-Spark @ 08ac8b3def ›
Image: ghcr.io/0xsero/deepseek-v4.1-flash-spark@sha256:3cbc8ec016f5fbfc82eba3480de12399cbce31e7b76aa3108b8fe8246c42a79d
Source: 0xSero/local-ai-images @ main/deepseek-v4.1-flash-spark ›
Launch: vllm-deepseek-v4.1-flash-spark-256k-2x ›
Recipe file: registry/recipes/nvidia/dgx-spark-gb10-128gb/deepseek-v4.1-flash.vllm.256k.2x.json ›
Model released: Sep 10, 2026
Tested: owner, 2x DGX Spark (GB10)