Gemma 4 12B

NVFP4 · sglang · 128K context · RTX 5090 · 32 GB

Google's Gemma 4 12B from May 2026: a compact multimodal model with thinking, good with images and documents.

– tok/sdecode

– tok/sprefill

128Kcontext window

Earlier checkearlier

Run it

Follow the steps below to download and run this recipe. Omarchy Local AI lists the recipes it can launch on your hardware. The server then answers on http://localhost:8000/v1.

1 Download the weights

hf download unsloth/gemma-4-12b-it-NVFP4 \
  --revision b1f649734b34aa5575b03d186abd1b9be3d0d5c4

2 Write the server config

cat > gemma-4-12b.sglang.128k.yml <<'EOF'
--- /tmp/gemma4_unified.py.orig	2026-08-23 20:25:26.703455135 +0000
+++ python/sglang/srt/models/gemma4_unified.py	2026-08-23 20:35:46.352246434 +0000
@@ -70,7 +70,12 @@

     def __init__(self, config):
         super().__init__()
-        patch_dim = config.model_patch_size**2 * 3  # 48*48*3 = 6912
+        model_patch_size = getattr(config, "model_patch_size", None)
+        if model_patch_size is None:
+            # NVFP4 republished configs carry patch_size*pooling_kernel_size
+            # (16*3=48) instead of the flattened model_patch_size field.
+            model_patch_size = config.patch_size * getattr(config, "pooling_kernel_size", 1)
+        patch_dim = model_patch_size**2 * 3  # 48*48*3 = 6912
         mm_embed_dim = config.mm_embed_dim

         self.patch_ln1 = nn.LayerNorm(patch_dim)
@@ -110,7 +115,11 @@

     def __init__(self, multimodal_config, text_config):
         super().__init__()
-        self.multimodal_hidden_size = multimodal_config.output_proj_dims
+        self.multimodal_hidden_size = getattr(
+            multimodal_config,
+            "audio_embed_dim",
+            multimodal_config.output_proj_dims,
+        )
         self.text_hidden_size = text_config.hidden_size
         self.embedding_pre_projection_norm = Gemma4RMSNorm(
             self.multimodal_hidden_size,
@@ -190,7 +199,8 @@
         )

         text_tie = getattr(text_config, "tie_word_embeddings", True)
-        if self.pp_group.world_size == 1 and text_tie:
+        self.lm_head_is_tied = self.pp_group.world_size == 1 and text_tie
+        if self.lm_head_is_tied:
             self.lm_head = self.language_model.embed_tokens
         elif self.pp_group.is_last_rank:
             self.lm_head = ParallelLMHead(
EOF

3 Start the server

docker run --rm \
  --gpus all \
  -p 8000:30000 \
  --shm-size 16g \
  -e HF_HOME=/root/.cache/huggingface \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v $PWD/gemma-4-12b.sglang.128k.yml:/tmp/gemma4-engine-patch.diff:ro \
  --entrypoint /bin/bash \
  lmsysorg/sglang:dev-cu13@sha256:6cd4635214f279e0a43019f88e3120d407567640a58aa7dcc0085e3d91402cc4 \
  -lc 'set -euo pipefail; echo '\''dede8848dbcbfcfc6507da963a65d120add93101b6ff5fb54a03f19e86233d11  /tmp/gemma4-engine-patch.diff'\'' | sha256sum -c -; echo '\''d5498b253f35e83ab0aaf219a2c2bf2f42c6bbd3f95ffce02760e78b2e38a4e9  /sgl-workspace/sglang/python/sglang/srt/models/gemma4_unified.py'\'' | sha256sum -c -; (cd /sgl-workspace/sglang && patch -p0 < /tmp/gemma4-engine-patch.diff); echo '\''b0614b99a0d7fe654ed102fb5db04c578e3be6042896f73f9418965e6672c737  /sgl-workspace/sglang/python/sglang/srt/models/gemma4_unified.py'\'' | sha256sum -c -; exec /opt/sglang/bin/python -m sglang.launch_server --model-path unsloth/gemma-4-12b-it-NVFP4 --revision b1f649734b34aa5575b03d186abd1b9be3d0d5c4 --tp 1 --host 0.0.0.0 --port 30000 --context-length 131072 --mem-fraction-static 0.8827875 --attention-backend triton --enable-cache-report --trust-remote-code --reasoning-parser gemma4 --tool-call-parser gemma4'

What it passed

✓ loadLoads and serves within an hour

✓ chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Passed the older check (loads and chats); a full six-check run is pending.

Details

Weights: unsloth/gemma-4-12b-it-NVFP4 @ b1f649734b ›
Image: lmsysorg/sglang:dev-cu13@sha256:6cd4635214f279e0a43019f88e3120d407567640a58aa7dcc0085e3d91402cc4
Launch: sglang-gemma-4-12b-nvfp4-128k ›
Recipe file: registry/recipes/nvidia/rtx-5090-32gb/gemma-4-12b.sglang.128k.json ›
Model released: May 23, 2026
Tested: earlier acceptance