Nemotron-Labs-3-Puzzle-75B-A9B

NVFP4 · vllm · 256K context · DGX Spark GB10 · 128 GB

NVIDIA's Nemotron Labs 3 Puzzle mixture-of-experts.

– tok/sdecode

– tok/sprefill

256Kcontext window

Reported by MiaAI-LabJul 8, 2026

Run it

This is the launch as MiaAI-Lab publishes it. Their repository has the full setup: MiaAI-Lab/Nemotron-Labs-3-Puzzle-75B-DGX-Spark@c77d26681086 › The server then answers on http://localhost:8000/v1.

1 Set up first

# Download the model on the host first (hf download nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4); HF_HUB_OFFLINE=1. start.sh picks the first snapshot dir under ~/.cache/huggingface/hub and replaces symlinked *.py/*.json/*.jinja in it with real files so trust_remote_code loads. Mounts ~/.cache/huggingface and ~/.cache/flashinfer -> /root/.cache/flashinfer. In-container prelude relinks aarch64 system libnccl and seds two FlashInfer compatibility fixes (all-reduce ImportError -> Exception, cuda_ipc skip stub libs).
# Full instructions: https://github.com/MiaAI-Lab/Nemotron-Labs-3-Puzzle-75B-DGX-Spark/tree/c77d26681086df72aadb6bf2879745408537e0b0

2 Download the weights

hf download nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 \
  --revision 1d370e47fbc56d1019a471c2339663cdbbb5236f

3 Start the server

docker run --rm \
  --gpus all \
  -p 8000:8888 \
  --gpus all \
  --privileged \
  --network host \
  --ipc host \
  --shm-size 10g \
  --memory 112g \
  --memory-swap 112g \
  --ulimit memlock=-1 \
  --ulimit nofile=1048576 \
  --shm-size 10g \
  -e FLASHINFER_CUDA_ARCH_LIST=12.1a \
  -e HF_HUB_OFFLINE=1 \
  -e LD_LIBRARY_PATH=/usr/local/cuda-13.0/targets/sbsa-linux/lib:/usr/local/lib/python3.12/dist-packages/nvidia/cu13/lib \
  -e LD_PRELOAD=/usr/local/cuda-13.0/targets/sbsa-linux/lib/libcudart.so.13 \
  -e MAX_JOBS=6 \
  -e PATH=/usr/local/cuda-13/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  -e TORCH_CUDA_ARCH_LIST=12.1a \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e VLLM_SKIP_INIT_MEMORY_CHECK=1 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint bash \
  vllm/vllm-openai:v0.24.0@sha256:251eba5cc7c12fed0b75da22a9240e582b1c9e39f6fbc064f86781b963bd814f \
  -lc 'rm -f /usr/local/lib/python3.12/dist-packages/nvidia/nccl/lib/libnccl.so.2 2>/dev/null
ln -sf /usr/lib/aarch64-linux-gnu/libnccl.so.2 /usr/local/lib/python3.12/dist-packages/nvidia/nccl/lib/libnccl.so.2 2>/dev/null
FAR=/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/flashinfer_all_reduce.py
sed -i '\''s/^except ImportError:/except Exception:/'\'' "$FAR" 2>/dev/null
CIPC=/usr/local/lib/python3.12/dist-packages/flashinfer/comm/cuda_ipc.py
sed -i '\''s/if lib_name in line:/if lib_name in line and "stub" not in line:/'\'' "$CIPC" 2>/dev/null
exec vllm serve ...' \
  serve \
  '/root/.cache/huggingface/hub/models--nvidia--NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4/snapshots/${SNAPSHOT}/' \
  --served-model-name Nemotron-75b-Puzzle \
  Nemotron-75b-Puzzle \
  --host 0.0.0.0 \
  --port 8888 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enable-expert-parallel \
  --mamba-backend flashinfer \
  --moe-backend FLASHINFER_CUTLASS \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --max-num-seqs 7 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.85 \
  --enforce-eager \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice \
  --enable-prefix-caching

What it passed

– loadLoads and serves within an hour

– chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by MiaAI-Lab in MiaAI-Lab/Nemotron-Labs-3-Puzzle-75B-DGX-Spark@c77d26681086. Our six checks have not run on it yet.

Details

Weights: nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 @ 1d370e47fb ›
Image: vllm/vllm-openai:v0.24.0@sha256:251eba5cc7c12fed0b75da22a9240e582b1c9e39f6fbc064f86781b963bd814f
Source: MiaAI-Lab/Nemotron-Labs-3-Puzzle-75B-DGX-Spark @ c77d2668108 ›
Launch: vllm-nemotron-labs-3-puzzle-75b-a9b-nvfp4-256k ›
Recipe file: registry/recipes/nvidia/dgx-spark-gb10-128gb/nemotron-labs-3-puzzle-75b-a9b.vllm.256k.json ›
Model released: Jun 24, 2026
Tested: not yet by us; reported by miaai-lab