Step-5-Preview

· vllm · 256K context · 4 × DGX Spark GB10 · 512 GB

StepFun's Step-5 preview, a ~600B mixture-of-experts model with image and video input.

21 tok/sdecode

888 tok/sprefill

256Kcontext window

Tested on this cardOct 6, 2026

Run it

Follow the steps below to download and run this recipe. Omarchy Local AI lists the recipes it can launch on your hardware. The server then answers on http://localhost:8000/v1.

1 Set up first

# Head (rank 0, API) + three workers (ranks 1-3, --headless, same args with their NODE_RANK) over the CX7 RoCE fabric (--nnodes 4). Set NODE_RANK, HEAD_IP, NODE_IP, CX7_IF and CX7_IB per node; NODE_IP must be the node's fabric IPv4 and NCCL_IB_HCA must list the RDMA device of the fabric interface first. Every node downloads the pinned 0xSero/Step-5-Preview-Spark repository (~342 GB, files under ~5 GiB, no reassembly) to local NVMe and mounts it read-only at /model. Mount a persistent /cache for compile and RoCE caches; clear /cache/vllm/torch_compile_cache after changing body format, MTP or graph settings. VLLM_DISABLE_SHARED_EXPERTS_STREAM=1 is required (a shared-expert side stream deadlocks the EXL3 MoE kernel). Run a host memory guard on every node (GB10 unified memory; github.com/0xSero/Step-5-Preview-Four-Sparks scripts/memguard.sh). Load ~12-13 minutes. KV 20 GB per rank = 999,279 tokens at 262,144 context. Validated: all six lab gates pass on four DGX Sparks (2026-10-06).
# Full instructions: https://github.com/0xSero/local-ai-images/tree/main/step-5-preview-spark

2 Download the weights

hf download 0xSero/Step-5-Preview-Spark \
  --revision 292721d6418e6d6d36a696b4e3efda28e394cab4 \
  --local-dir ~/models/Step-5-Preview-Spark-292721d6

3 Start the server on each of the 4 machines (NODE_RANK 0 to 3)

docker run --rm \
  --gpus all \
  -p 8000:8000 \
  --gpus all \
  --network host \
  --ipc=host \
  --privileged \
  --shm-size 32g \
  --ulimit memlock=-1 \
  --ulimit nofile=1048576:1048576 \
  --device /dev/infiniband \
  --shm-size 32g \
  -e B12X_COMPILE_CACHE_DIR=/cache/b12x \
  -e B12X_ROCE_CACHE_DIR=/cache/b12x-roce \
  -e B12X_ROCE_TRAFFIC_CLASS=106 \
  -e CUDA_VISIBLE_DEVICES=0 \
  -e EXL3_INT8_GEMV=1 \
  -e GLOO_SOCKET_IFNAME="${CX7_IF}" \
  -e HF_HUB_OFFLINE=1 \
  -e MALLOC_ARENA_MAX=2 \
  -e NCCL_DEBUG=WARN \
  -e NCCL_IB_DISABLE=0 \
  -e NCCL_IB_GID_INDEX=3 \
  -e NCCL_IB_HCA="${CX7_IB}" \
  -e NCCL_IB_MERGE_NICS=1 \
  -e NCCL_IB_TC=106 \
  -e NCCL_NET_PLUGIN=none \
  -e NCCL_PROTO=Simple \
  -e NCCL_RAS_ENABLE=0 \
  -e NCCL_SOCKET_IFNAME="${CX7_IF}" \
  -e OMP_NUM_THREADS=16 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e STEP5_BODY_FORMAT=hybrid \
  -e STEP5_MODEL_DIR=/model \
  -e ST_EXL3_PREFILL=st \
  -e ST_EXL3_PREFILL_MIN_ROWS=1 \
  -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e VLLM_DISABLE_SHARED_EXPERTS_STREAM=1 \
  -e VLLM_ENABLE_PCIE_ALLREDUCE=0 \
  -e VLLM_ENABLE_ROCE_ALLREDUCE=1 \
  -e VLLM_HOST_IP="${NODE_IP}" \
  -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 \
  -e VLLM_ROCE_ALLGATHER_MAX_SIZE=16MB \
  -e VLLM_ROCE_ALLREDUCE_MAX_SIZE=2MB \
  -e VLLM_USE_BREAKABLE_CUDAGRAPH=0 \
  -e VLLM_USE_V2_MODEL_RUNNER=0 \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e XDG_CACHE_HOME=/cache \
  -v ~/models/Step-5-Preview-Spark-292721d6:/model:ro \
  --entrypoint vllm \
  ghcr.io/0xsero/step-5-preview-spark@sha256:4e28f849a414a483b44be50a09198d76cbc859096b354796dc25ee2159e05a5e \
  serve \
  /model \
  --served-model-name step-5-preview-spark \
  --dtype float16 \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 4 \
  --nnodes 4 \
  --node-rank "${NODE_RANK}" \
  --master-addr "${HEAD_IP}" \
  --master-port 29665 \
  --kv-cache-memory-bytes 20000000000 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 4096 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --safetensors-load-strategy lazy \
  --limit-mm-per-prompt '{"image":4,"video":1}' \
  --generation-config vllm \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,8,12]}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --reasoning-parser step3p5 \
  --tool-call-parser step3p5 \
  --enable-auto-tool-choice \
  --enable-prompt-tokens-details \
  --enable-force-include-usage

What it passed

✓ loadLoads and serves within an hour

✓ chatAnswers a plain question and stops on its own

✓ reasoningThinks separately and gets 17 × 23 right

✓ toolsCalls a tool with the right arguments and uses the result

✓ contextRecalls a code buried in a prompt that fills 85% of the window

✓ speedDecodes at 15 tok/s or more

Passed all six checks on a real 4x DGX Spark (GB10) on Oct 6, 2026.

Details

Weights: 0xSero/Step-5-Preview-Spark @ 292721d641 ›
Image: ghcr.io/0xsero/step-5-preview-spark@sha256:4e28f849a414a483b44be50a09198d76cbc859096b354796dc25ee2159e05a5e
Source: 0xSero/local-ai-images @ main/step-5-preview-spark ›
Launch: vllm-step-5-preview-spark-256k-4x ›
Recipe file: registry/recipes/nvidia/dgx-spark-gb10-128gb/step-5-preview.vllm.256k.4x.json ›
Model released: Sep 20, 2026
Tested: owner, 4x DGX Spark (GB10)