Gemma 4 31B

NVFP4 · vllm · 256K context · DGX Spark GB10 · 128 GB

Google's dense Gemma 4, 31B parameters.

– tok/sdecode

– tok/sprefill

256Kcontext window

Reported by MiaAI-LabJul 16, 2026

Run it

This is the launch as MiaAI-Lab publishes it. Their repository has the full setup: MiaAI-Lab/Gemma-4-31B-IT-NVFP4-Recipe@ac429cca70ae › The server then answers on http://localhost:8000/v1.

1 Set up first

# Docker. start.sh pre-downloads both gated models (hf / huggingface-cli / fallback python in the vLLM image) into ~/.cache/huggingface; HF_TOKEN required with access to both repos. Mounts repo's Gemma 4 canonical chat_template.jinja (2026-07-09).
# Full instructions: https://github.com/MiaAI-Lab/Gemma-4-31B-IT-NVFP4-Recipe/tree/ac429cca70aedfa4afb446cefc5f5f1bfa1e8802

2 Download the weights

hf download nvidia/Gemma-4-31B-IT-NVFP4 \
  --revision 4135a98a9b728a548947683219633b25682223ac

3 Start the server

docker run --rm \
  --gpus all \
  -p 8000:8888 \
  --network host \
  --ipc host \
  --gpus all \
  -v ${TRITON_CACHE_DIR}:/root/.triton \
  -v ${WORK_DIR}/chat_template.jinja:/workspace/chat_template.jinja \
  -v ${WORK_DIR}:/workspace \
  -e HF_HOME="${HF_HOME}" \
  -e HF_TOKEN="${HF_TOKEN}" \
  -e TRITON_CACHE_DIR=/root/.triton \
  -e VLLM_TARGET_DEVICE=cuda \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.25.1@sha256:e4f88a835143cd22aee2397a26ec6bb80b3a4a6fe0c882bcbc63822904766089 \
  nvidia/Gemma-4-31B-IT-NVFP4 \
  --host 0.0.0.0 \
  --port 8888 \
  --tensor-parallel-size 1 \
  --trust-remote-code \
  --quantization modelopt \
  --attention-backend triton_attn \
  --chat-template /workspace/chat_template.jinja \
  --gpu-memory-utilization 0.70 \
  --max-model-len 262144 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 8192 \
  --kv-cache-dtype fp8 \
  --enable-chunked-prefill \
  --async-scheduling \
  --mm-encoder-tp-mode data \
  --enable-prefix-caching \
  --allowed-media-domains '*' \
  --speculative-config '{"method":"mtp","model":"google/gemma-4-31B-it-assistant","num_speculative_tokens":4}' \
  --load-format fastsafetensors \
  --limit-mm-per-prompt '{"image": 4, "video": 1, "audio": 1}' \
  --reasoning-parser gemma4 \
  --override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' \
  --default-chat-template-kwargs '{"enable_thinking":true}' \
  --tool-call-parser gemma4 \
  --enable-auto-tool-choice

What it passed

– loadLoads and serves within an hour

– chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by MiaAI-Lab in MiaAI-Lab/Gemma-4-31B-IT-NVFP4-Recipe@ac429cca70ae. Our six checks have not run on it yet.

Details

Weights: nvidia/Gemma-4-31B-IT-NVFP4 @ 4135a98a9b ›
Image: vllm/vllm-openai:v0.25.1@sha256:e4f88a835143cd22aee2397a26ec6bb80b3a4a6fe0c882bcbc63822904766089
Source: MiaAI-Lab/Gemma-4-31B-IT-NVFP4-Recipe @ ac429cca70aedfa4afb4 ›
Launch: vllm-gemma-4-31b-nvfp4-256k ›
Recipe file: registry/recipes/nvidia/dgx-spark-gb10-128gb/gemma-4-31b.vllm.256k.json ›
Model released: Mar 11, 2026
Tested: not yet by us; reported by miaai-lab