MiMo-V2.5

NVFP4 · vllm · 977K context · 2 × DGX Spark GB10 · 256 GB

Xiaomi's MiMo V2.5 omni model.

30 tok/sdecode

– tok/sprefill

977Kcontext window

Reported by MiaAI-LabJul 15, 2026

Run it

This is the launch as MiaAI-Lab publishes it. Their repository has the full setup: MiaAI-Lab/MiMo-V2.5-vLLM-Dual-DGX-Sparks@eb9c5fa61ed1 › The server then answers on http://localhost:8000/v1.

1 Set up first

# Containers started with 'sleep infinity' on both nodes (models--lukealonso--MiMo-V2.5-NVFP4 hub dir mounted ro at /model_cache, repo dir at /workspace; worker gets an rsync copy of the repo). Then docker exec: recipe/apply-mods.sh (7 mods: drop-caches, ray-keep-node-nccl-hca, fix-prometheus-instrumentator-router, fix-mimo-v2-vllm, fix-mimo-mtp-patches, fix-modelopt-mixed-mxfp8, nvfp4-kv-diffkv) and 10 engine patches from patches/ on both ranks; Ray head (port 6379, --num-gpus=1 --object-store-memory=1073741824, after unsetting RAY_OVERRIDE_NODE_IP_ADDRESS/RAY_NODE_IP_ADDRESS) and worker join; wait for 2/2 GPUs; then on the head: source recipe/env.sh plus overrides and nohup bash recipe/launch-omni.sh. Weights downloaded with hf on the head at the pinned revision and rsynced to the worker. Passwordless SSH to zurih@10.0.0.2.
# Full instructions: https://github.com/MiaAI-Lab/MiMo-V2.5-vLLM-Dual-DGX-Sparks/tree/eb9c5fa61ed183a085f622dbb4c67f9fcb065a47

2 Download the weights

hf download lukealonso/MiMo-V2.5-NVFP4 \
  --revision a147dd04d6cf861e43b2d783dcde23b53ab7ee68 \
  --local-dir ~/models/MiMo-V2.5-NVFP4-a147dd04

3 Start the server on each of the 2 machines (NODE_RANK 0 to 1)

docker run --rm \
  --gpus all \
  -p 8000:8888 \
  --gpus all \
  --network host \
  --ipc host \
  --shm-size 16g \
  --workdir /workspace \
  --device /dev/infiniband:/dev/infiniband \
  --ulimit memlock=-1:-1 \
  --ulimit stack=67108864 \
  --shm-size 16g \
  -e API_PORT=8888 \
  -e ATTENTION_BACKEND=triton_attn_diffkv \
  -e BLOCK_SIZE=64 \
  -e DEFAULT_TEMPERATURE=0 \
  -e DEFAULT_TOP_P=0.95 \
  -e ENABLE_MTP=1 \
  -e ENFORCE_EAGER=1 \
  -e GLOO_SOCKET_IFNAME=enp1s0f1np1 \
  -e GPU_MEMORY_UTILIZATION=0.83 \
  -e HEAD_ROCE_IP=10.0.0.1 \
  -e KV_CACHE_DTYPE=nvfp4 \
  -e LOAD_FORMAT=safetensors \
  -e MAX_MODEL_LEN=1000000 \
  -e MAX_NUM_BATCHED_TOKENS=2048 \
  -e MAX_NUM_SEQS=3 \
  -e MAX_TOKENS_LIMIT=32768 \
  -e MOE_BACKEND=flashinfer_cutlass \
  -e MTP_SPEC_TOKENS=1 \
  -e NCCL_CROSS_NIC=1 \
  -e NCCL_CUMEM_ENABLE=0 \
  -e NCCL_IB_DISABLE=0 \
  -e NCCL_IB_GID_INDEX=3 \
  -e NCCL_IB_HCA=rocep1s0f1 \
  -e NCCL_IB_MERGE_NICS=0 \
  -e NCCL_IB_SUBNET_AWARE_ROUTING=1 \
  -e NCCL_MAX_NCHANNELS=2 \
  -e NCCL_NET=IB \
  -e NCCL_NET_GDR_LEVEL=LOC \
  -e NCCL_NET_PLUGIN=none \
  -e NCCL_NVLS_ENABLE=0 \
  -e NCCL_PROTO=LL \
  -e NCCL_SOCKET_FAMILY=AF_INET \
  -e NCCL_SOCKET_IFNAME=enp1s0f1np1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e RAY_ADDRESS=10.0.0.1:6379 \
  -e RAY_TMPDIR=/dev/shm/ray \
  -e RAY_memory_monitor_refresh_ms=0 \
  -e REPETITION_PENALTY=1.08 \
  -e TENSOR_PARALLEL_SIZE=2 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e VLLM_FLASHINFER_MOE_BACKEND=throughput \
  -e VLLM_HOST_IP="${NODE_ROCE_IP}" \
  -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \
  -e VLLM_MIMO_MTP1_GREEDY_FAST=1 \
  -e VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass \
  -e VLLM_NVFP4_INLINE=1 \
  -e VLLM_USE_FLASHINFER_MOE_FP4=1 \
  -e VLLM_USE_RAY_COMPILED_DAG_OVERLAP_COMM=0 \
  -e VLLM_USE_RAY_V2_EXECUTOR_BACKEND=0 \
  -e VLLM_WMMA_AUTHCOMPARE=0 \
  -e VLLM_WMMA_COMPARE=0 \
  -e VLLM_WMMA_DECODE=1 \
  -e VLLM_WMMA_INSPECT=0 \
  -v ~/models/MiMo-V2.5-NVFP4-a147dd04:/model_cache:ro \
  --entrypoint sleep \
  ghcr.io/miaai-lab/mimo-v2.5-vllm-dual-dgx-sparks:20260704@sha256:e37e2972f3e8eec2b326767293a99e61cbdc11c1551e2c481c7c49f5f2a31809 \
  infinity \
  serve \
  /model_cache/snapshots/a147dd04d6cf861e43b2d783dcde23b53ab7ee68 \
  --served-model-name MiMo-V2.5-NVFP4 \
  --trust-remote-code \
  --dtype auto \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --distributed-executor-backend ray \
  --load-format safetensors \
  --hf-overrides '{"architectures":["MiMoV2OmniForCausalLM"]}' \
  --limit-mm-per-prompt '{"image":4,"video":1,"audio":1}' \
  --mm-encoder-tp-mode data \
  --attention-backend triton_attn_diffkv \
  --moe-backend flashinfer_cutlass \
  --kv-cache-dtype nvfp4 \
  --gpu-memory-utilization 0.83 \
  --max-model-len 1000000 \
  --max-num-batched-tokens 2048 \
  --max-num-seqs 3 \
  --block-size 64 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --no-async-scheduling \
  --enable-auto-tool-choice \
  --tool-call-parser mimo \
  --reasoning-parser mimo \
  --default-chat-template-kwargs '{"enable_thinking":false}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1,"use_local_argmax_reduction":false}' \
  --generation-config vllm \
  --enforce-eager \
  --disable-uvicorn-access-log \
  --host 0.0.0.0 \
  --port 8888

What it passed

– loadLoads and serves within an hour

– chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by MiaAI-Lab in MiaAI-Lab/MiMo-V2.5-vLLM-Dual-DGX-Sparks@eb9c5fa61ed1, where it reports 30 tok/s. Our six checks have not run on it yet.

Details

Weights: lukealonso/MiMo-V2.5-NVFP4 @ a147dd04d6 ›
Image: ghcr.io/miaai-lab/mimo-v2.5-vllm-dual-dgx-sparks:20260704@sha256:e37e2972f3e8eec2b326767293a99e61cbdc11c1551e2c481c7c49f5f2a31809
Source: MiaAI-Lab/MiMo-V2.5-vLLM-Dual-DGX-Sparks @ eb9c5fa61ed183a08 ›
Launch: vllm-mimo-v2.5-nvfp4-976k-2x ›
Recipe file: registry/recipes/nvidia/dgx-spark-gb10-128gb/mimo-v2.5.vllm.976k.2x.json ›
Model released: Apr 27, 2026
Tested: not yet by us; reported by miaai-lab