gemma-4-31B-it-AWQ-4bit

· vllm · 256K context · DGX Spark GB10 · 128 GB

cyankiwi's AWQ INT4 quant of Gemma 4 31B.

22 tok/sdecode

98 tok/sprefill

256Kcontext window

Reported by toncaoSep 30, 2026

Run it

This is the launch as toncao publishes it. Their repository has the full setup: toncao/local-ai-registry@f5be8bf9c62b449a7fe1c82ce01f9bd10426e861 › The server then answers on http://localhost:8000/v1.

1 Download the weights

hf download cyankiwi/gemma-4-31B-it-AWQ-4bit \
  --revision 6f1b616c64bca2a05d67fdd6437c5500a8509327 \
  --local-dir ~/models/gemma-4-31B-it-AWQ-4bit-6f1b616c

2 Start the server

docker run --rm \
  --gpus all \
  -p 8000:8000 \
  --shm-size 16g \
  -v ~/models/gemma-4-31B-it-AWQ-4bit-6f1b616c:/models/gemma-4-31B-it-AWQ-4bit-6f1b616c:ro \
  --entrypoint vllm \
  vllm/vllm-openai:v0.30.0@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90 \
  serve \
  /models/gemma-4-31B-it-AWQ-4bit-6f1b616c \
  --served-model-name gemma-4-31B-it-AWQ-4bit-6f1b616c \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.8 \
  --kv-cache-dtype fp8 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --default-chat-template-kwargs '{"enable_thinking":true}' \
  --speculative-config '{"method":"mtp","model":"google/gemma-4-31B-it-assistant","revision":"627c5ec1458b9086b841a91e0512fd31fd2fbbf1","num_speculative_tokens":4}' \
  --safetensors-load-strategy eager

What it passed

– loadLoads and serves within an hour

– chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by toncao in toncao/local-ai-registry@f5be8bf9c62b449a7fe1c82ce01f9bd10426e861, where it reports 22.2 tok/s. Our six checks have not run on it yet. Reported decode averages the entire answer; requests omitted temperature. Canonical lab acceptance is pending.

Details

Weights: cyankiwi/gemma-4-31B-it-AWQ-4bit @ 6f1b616c64 ›
Image: vllm/vllm-openai:v0.30.0@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90
Source: toncao/local-ai-registry @ f5be8bf9c62b449a7fe1c82ce01f9bd10 ›
Engine profile: vllm ›
Recipe file: registry/recipes/nvidia/dgx-spark-gb10-128gb/gemma-4-31b-it-awq-4bit.vllm.256k.json ›
Model released: Apr 2, 2026
Tested: not yet by us; reported by toncao