Gemma 4 26B A4B

NVFP4 · vllm · 256K context · DGX Spark GB10 · 128 GB

Google's Gemma 4 mixture-of-experts model: 26B total, 4B active, so it runs fast while seeing images and thinking.

30 tok/sdecode

– tok/sprefill

256Kcontext window

Reported by MiaAI-LabJul 5, 2026

Run it

This is the launch as MiaAI-Lab publishes it. Their repository has the full setup: MiaAI-Lab/Gemma-4-26B-A4B-DGX-Spark-18-concurrencies@39e94192bd6f › The server then answers on http://localhost:8000/v1.

1 Set up first

# Mounts a patched gemma4.py over /usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/gemma4.py:ro (required, start.sh aborts without it). Mounts ~/.cache/huggingface, ./.cache/triton -> /tmp/triton-cache, ./.cache/torchinductor -> /tmp/torchinductor-cache, ./.cache/xdg -> /tmp/.cache (chmod 777), and repo -> /workspace. Uses the image's default entrypoint (vllm serve).
# Full instructions: https://github.com/MiaAI-Lab/Gemma-4-26B-A4B-DGX-Spark-18-concurrencies/tree/39e94192bd6f114a222d434cf2bf4dcf173248a5

2 Download the weights

hf download nvidia/Gemma-4-26B-A4B-NVFP4 \
  --revision a19cfe00be84568a6867111c9a68c9c44fdcffe6

3 Start the server

docker run --rm \
  --gpus all \
  -p 8000:8888 \
  --network host \
  --ipc host \
  --gpus device=0 \
  -e HF_HOME=/root/.cache/huggingface \
  -e HF_TOKEN='${HF_TOKEN:-}' \
  -e TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor-cache \
  -e TRITON_CACHE_DIR=/tmp/triton-cache \
  -e VLLM_CACHE_ROOT=/tmp/.cache/vllm \
  -e VLLM_TARGET_DEVICE=cuda \
  -e XDG_CACHE_HOME=/tmp/.cache \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:nightly@sha256:1b88c3afc77c73a3858730254d6eaf73fd7ff6e226ced60f069d9277fd8d6b2b \
  nvidia/Gemma-4-26B-A4B-NVFP4 \
  --host 0.0.0.0 \
  --port 8888 \
  --tensor-parallel-size 1 \
  --trust-remote-code \
  --quantization modelopt \
  --gpu-memory-utilization 0.70 \
  --max-model-len 262144 \
  --max-num-seqs 18 \
  --max-num-batched-tokens 8192 \
  --kv-cache-dtype fp8 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --attention-backend triton_attn \
  --load-format fastsafetensors \
  --reasoning-parser gemma4 \
  --tool-call-parser gemma4 \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"enable_thinking":1}' \
  --limit-mm-per-prompt '{"image": 1}' \
  --allowed-media-domains '*' \
  --mm-encoder-tp-mode weights \
  --override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":64,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' \
  --speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":1}'

What it passed

– loadLoads and serves within an hour

– chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by MiaAI-Lab in MiaAI-Lab/Gemma-4-26B-A4B-DGX-Spark-18-concurrencies@39e94192bd6f, where it reports 29.77 tok/s. Our six checks have not run on it yet.

Details

Weights: nvidia/Gemma-4-26B-A4B-NVFP4 @ a19cfe00be ›
Image: vllm/vllm-openai:nightly@sha256:1b88c3afc77c73a3858730254d6eaf73fd7ff6e226ced60f069d9277fd8d6b2b
Source: MiaAI-Lab/Gemma-4-26B-A4B-DGX-Spark-18-concurrencies @ 39e94 ›
Launch: vllm-gemma-4-26b-a4b-nvfp4-256k ›
Recipe file: registry/recipes/nvidia/dgx-spark-gb10-128gb/gemma-4-26b-a4b.vllm.256k.json ›
Model released: Mar 12, 2026
Tested: not yet by us; reported by miaai-lab