GLM-5.2

NVFP4 · vllm · 230K context · 3 × DGX Spark GB10 · 384 GB

Z.ai's GLM-5.2 flagship, here as an NVFP4 + AQLM hybrid.

26 tok/sdecode

– tok/sprefill

230Kcontext window

Reported by MiaAI-LabAug 3, 2026

Run it

This is the launch as MiaAI-Lab publishes it. Their repository has the full setup: MiaAI-Lab/GLM-5.2-NVFP4-AQLM-Triple-DGX-Sparks@5c85163ccb8d › The server then answers on http://localhost:8000/v1.

1 Set up first

# 3 Sparks on ConnectX-7 RoCE (pairwise triangle), SSH key from head to both workers, pip install pexpect on head, host NCCL 2.30.7 at $HOME/nccl-2.30.7 mounted at /nccl. ./start.sh download (hf download at HF_REVISION) + sync (rsync ~272 GB to workers) + pull (docker save/rsync/load image to workers) + ray (Ray head container on head, Ray worker containers on workers via ssh, each `bash -lc 'cd /opt/vllm && source .venv/bin/activate && ray start ...'`) + serve (docker exec vllm serve inside Ray head container, log /tmp/vllm-serve.log). Disable earlyoom on all nodes.
# Full instructions: https://github.com/MiaAI-Lab/GLM-5.2-NVFP4-AQLM-Triple-DGX-Sparks/tree/5c85163ccb8d98d395880e71e2dbd03976a3f4ad

2 Download the weights

hf download jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid \
  --revision 53e0082eedebd806b63e19779c47905937d768ca \
  --local-dir ~/models/GLM-5.2-NVFP4-AQLM-hybrid-53e0082e

3 Start the server on each of the 3 machines (NODE_RANK 0 to 2)

docker run --rm \
  --gpus all \
  -p 8000:8888 \
  --network host \
  --ipc host \
  --privileged \
  --cap-add IPC_LOCK \
  --gpus all \
  --ulimit memlock=-1:-1 \
  --ulimit stack=67108864 \
  --device /dev/infiniband:/dev/infiniband \
  -v ${NCCL_HOST_DIR}:/nccl:ro \
  --entrypoint bash \
  --shm-size 10gb \
  -e CUDA_DEVICE_MAX_CONNECTIONS=32 \
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -e CUTE_DSL_ARCH=sm_121a \
  -e FLASHINFER_DISABLE_VERSION_CHECK=1 \
  -e GLM_MOE_AQLM_CB=l1 \
  -e GLM_MOE_AQLM_STREAM=1 \
  -e GLM_MOE_LANE_ROWS=1 \
  -e GLM_NVFP4_LUT256=1 \
  -e GLM_NVFP4_STREAM=1 \
  -e GLM_SHARED_EXPERTS_DEBUG=1 \
  -e GLOO_SOCKET_IFNAME=enP7s7 \
  -e HF_OVERRIDES='{"num_experts_per_tok":4,"index_topk_freq":8}' \
  -e HOST_IP="${NODE_IP}" \
  -e MODEL_DIR=/models/1m \
  -e NCCL_BUFFSIZE=1048576 \
  -e NCCL_CROSS_NIC=1 \
  -e NCCL_CUMEM_ENABLE=0 \
  -e NCCL_DEBUG=WARN \
  -e NCCL_GIN_ENABLE=0 \
  -e NCCL_GRAPH_MIXING_SUPPORT=1 \
  -e NCCL_IB_DISABLE=0 \
  -e NCCL_IB_GID_INDEX="${GID}" \
  -e NCCL_IB_HCA=rocep1s0f0,rocep1s0f1 \
  -e NCCL_IB_MERGE_NICS=0 \
  -e NCCL_IB_SUBNET_AWARE_ROUTING=1 \
  -e NCCL_IGNORE_CPU_AFFINITY=1 \
  -e NCCL_MAX_NCHANNELS=4 \
  -e NCCL_MIN_NCHANNELS=4 \
  -e NCCL_NET=IB \
  -e NCCL_P2P_DISABLE=1 \
  -e NCCL_PROTO=LL,LL128,Simple \
  -e NCCL_SHM_DISABLE=1 \
  -e NCCL_SOCKET_IFNAME=enP7s7 \
  -e OMP_NUM_THREADS=16 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e RAY_ADDRESS='${HEAD_IP}:6379' \
  -e RAY_memory_monitor_refresh_ms=0 \
  -e RAY_memory_usage_threshold=0.99 \
  -e SAFETENSORS_FAST_GPU=1 \
  -e TORCH_CUDA_ARCH_LIST=12.1a \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e VLLM_DISABLE_FP8_W8A16=0 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_GLM_TP_PAD=1 \
  -e VLLM_HOST_IP="${NODE_IP}" \
  -e VLLM_MARLIN_USE_ATOMIC_ADD=1 \
  -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \
  -e VLLM_MTP_INDEX_SHARE=1 \
  -e VLLM_NCCL_SO_PATH=/nccl/libnccl.so.2.30.7 \
  -e VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=192 \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -v ~/models/GLM-5.2-NVFP4-AQLM-hybrid-53e0082e:/models/1m:ro \
  --entrypoint bash \
  ghcr.io/miaai-lab/glm-5.2-nvfp4-triple-dgx-sparks:k12l1-vision@sha256:f8f350d46b33858eda4f0c0a5c39a7f0b27005111d393098baaaacae18c10fdb \
  -lc 'cd /opt/vllm && source .venv/bin/activate && vllm serve ...' \
  serve \
  /models/1m \
  --served-model-name glm-5.2 \
  --host 0.0.0.0 \
  --port 8888 \
  --trust-remote-code \
  --distributed-executor-backend ray \
  --tensor-parallel-size 3 \
  --pipeline-parallel-size 1 \
  --decode-context-parallel-size 1 \
  --dcp-comm-backend ag_rs \
  --kv-cache-dtype fp8_ds_mla \
  --gpu-memory-utilization 0.9 \
  --max-model-len 235392 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --no-enable-flashinfer-autotune \
  --compilation-config '{"mode":3,"cudagraph_mode":"FULL","cudagraph_capture_sizes":[4,8,12,16,20,24],"pass_config":{"fuse_allreduce_rms":true,"fuse_norm_quant":true}}' \
  --enable-auto-tool-choice \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --default-chat-template-kwargs '{"enable_thinking":false}' \
  --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":3}' \
  --async-scheduling \
  --hf-overrides '{"num_experts_per_tok":4,"index_topk_freq":8}' \
  --kv-cache-memory-bytes 12884901888 \
  --mm-encoder-tp-mode data

What it passed

– loadLoads and serves within an hour

– chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by MiaAI-Lab in MiaAI-Lab/GLM-5.2-NVFP4-AQLM-Triple-DGX-Sparks@5c85163ccb8d, where it reports 25.5 tok/s. Our six checks have not run on it yet.

Details

Weights: jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid @ 53e0082eed ›
Image: ghcr.io/miaai-lab/glm-5.2-nvfp4-triple-dgx-sparks:k12l1-vision@sha256:f8f350d46b33858eda4f0c0a5c39a7f0b27005111d393098baaaacae18c10fdb
Source: MiaAI-Lab/GLM-5.2-NVFP4-AQLM-Triple-DGX-Sparks @ 5c85163ccb8 ›
Launch: vllm-glm-5.2-other-229k-3x ›
Recipe file: registry/recipes/nvidia/dgx-spark-gb10-128gb/glm-5.2.vllm.229k.3x.json ›
Model released: Jun 16, 2026
Tested: not yet by us; reported by miaai-lab