gemma-4-12B-it-AWQ-INT4

· vllm · 128K context · DGX Spark GB10 · 128 GB

cyankiwi's AWQ INT4 quant of Gemma 4 12B.

43 tok/sdecode

843 tok/sprefill

128Kcontext window

Reported by toncaoSep 30, 2026

Run it

This is the launch as toncao publishes it. Their repository has the full setup: toncao/local-ai-registry@f5be8bf9c62b449a7fe1c82ce01f9bd10426e861 › The server then answers on http://localhost:8000/v1.

1 Download the weights

hf download cyankiwi/gemma-4-12B-it-AWQ-INT4 \
  --revision ad2d7e521f3eaee492973ab130025e610b216d2c \
  --local-dir ~/models/gemma-4-12B-it-AWQ-INT4-ad2d7e52

2 Start the server

docker run --rm \
  --gpus all \
  -p 8000:8000 \
  --shm-size 16g \
  -v ~/models/gemma-4-12B-it-AWQ-INT4-ad2d7e52:/models/gemma-4-12B-it-AWQ-INT4-ad2d7e52:ro \
  --entrypoint vllm \
  vllm/vllm-openai:v0.30.0@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90 \
  serve \
  /models/gemma-4-12B-it-AWQ-INT4-ad2d7e52 \
  --served-model-name gemma-4-12B-it-AWQ-INT4-ad2d7e52 \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 131072 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.8 \
  --kv-cache-dtype fp8 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --default-chat-template-kwargs '{"enable_thinking":true}' \
  --speculative-config '{"method":"mtp","model":"google/gemma-4-12B-it-assistant","revision":"46d4c6f13f0ac0ad827b915669b8df9b81c64c51","num_speculative_tokens":3}' \
  --safetensors-load-strategy eager

What it passed

– loadLoads and serves within an hour

– chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by toncao in toncao/local-ai-registry@f5be8bf9c62b449a7fe1c82ce01f9bd10426e861, where it reports 43.1 tok/s. Our six checks have not run on it yet. Reported decode averages the entire answer; requests omitted temperature. Canonical lab acceptance is pending.

Details

Weights: cyankiwi/gemma-4-12B-it-AWQ-INT4 @ ad2d7e521f ›
Image: vllm/vllm-openai:v0.30.0@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90
Source: toncao/local-ai-registry @ f5be8bf9c62b449a7fe1c82ce01f9bd10 ›
Engine profile: vllm ›
Recipe file: registry/recipes/nvidia/dgx-spark-gb10-128gb/gemma-4-12b-it-awq-int4.vllm.128k.json ›
Model released: Jun 4, 2026
Tested: not yet by us; reported by toncao