gemma-4-26B-A4B-it-AWQ-8bit

· vllm · 256K context · DGX Spark GB10 · 128 GB

cyankiwi's AWQ INT8 quant of Gemma 4 26B A4B.

49 tok/sdecode

401 tok/sprefill

256Kcontext window

Reported by toncaoSep 30, 2026

Run it

This is the launch as toncao publishes it. Their repository has the full setup: toncao/local-ai-registry@f5be8bf9c62b449a7fe1c82ce01f9bd10426e861 › The server then answers on http://localhost:8000/v1.

1 Download the weights

hf download cyankiwi/gemma-4-26B-A4B-it-AWQ-8bit \
  --revision 64cb43bd9e7b7e5966bfd557e5cb528ad0b1ec49 \
  --local-dir ~/models/gemma-4-26B-A4B-it-AWQ-8bit-64cb43bd

2 Start the server

docker run --rm \
  --gpus all \
  -p 8000:8000 \
  --shm-size 16g \
  -v ~/models/gemma-4-26B-A4B-it-AWQ-8bit-64cb43bd:/models/gemma-4-26B-A4B-it-AWQ-8bit-64cb43bd:ro \
  --entrypoint vllm \
  vllm/vllm-openai:v0.30.0@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90 \
  serve \
  /models/gemma-4-26B-A4B-it-AWQ-8bit-64cb43bd \
  --served-model-name gemma-4-26B-A4B-it-AWQ-8bit-64cb43bd \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.8 \
  --kv-cache-dtype fp8 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --default-chat-template-kwargs '{"enable_thinking":true}' \
  --speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-assistant","revision":"6e5aaaf4c42b98394530b8fda2e95cadd65c151c","num_speculative_tokens":2}' \
  --safetensors-load-strategy eager

What it passed

– loadLoads and serves within an hour

– chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by toncao in toncao/local-ai-registry@f5be8bf9c62b449a7fe1c82ce01f9bd10426e861, where it reports 49 tok/s. Our six checks have not run on it yet. Reported decode averages the entire answer; requests omitted temperature. Canonical lab acceptance is pending.

Details

Weights: cyankiwi/gemma-4-26B-A4B-it-AWQ-8bit @ 64cb43bd9e ›
Image: vllm/vllm-openai:v0.30.0@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90
Source: toncao/local-ai-registry @ f5be8bf9c62b449a7fe1c82ce01f9bd10 ›
Engine profile: vllm ›
Recipe file: registry/recipes/nvidia/dgx-spark-gb10-128gb/gemma-4-26b-a4b-it-awq-8bit.vllm.256k.json ›
Model released: Apr 3, 2026
Tested: not yet by us; reported by toncao