GLM-5.3-Flash

EXL3 3.05 bpw · exllamav3-nvme16g · 128K context · RTX 3090 · 24 GB

Z.ai's GLM-5.3-Flash, a large mixture-of-experts model for multi-GPU machines.

15 tok/sdecode

1,010 tok/sprefill

128Kcontext window

Reported by ownerOct 9, 2026

Run it

This is the launch as owner publishes it. Their repository has the full setup: lab/runs/rtx-3090-24gb.glm-5.3-flash.exllamav3-glm-5.3-flash-exl3-3.05bpw-nvme-16g-128k-rtx-3090-24gb.128k.20261009T124547.json › The server then answers on http://localhost:8000/v1.

1 Set up first

# NEEDS: 1x RTX 3090 24 GB (expert cache capped at 1,376 slots by GLM53_EC_MAX_SLOTS so a desktop on the same GPU cannot run the CUDA graphs out of memory); the container is capped at 16 GiB of host RAM (resources: --memory 16g --memory-swap 16g, swap off inside the cap) and runs with --ulimit memlock=-1 (the RAM tier is cudaHostRegister'ed). Below a 24 GiB cap the image's entrypoint (v4.2-nvme) applies a small-host budget: -rcs 1 (exllamav3 host recurrent-state cache 1 GiB instead of 4), GLM53_NV_MARGIN_GB=1.5, GLM53_NV_PF_RING=128 prefill read-ahead slots, and a deeper NVMe reader pool (GLM53_NV_THREADS=48 x GLM53_NV_PIECE_KB=1024), each overridable; NVMe modes are refused only under 15 GiB. The RAM tier sizes itself from the cap (970 experts = 8.5 GiB on the measured box); measured 13.6 GiB at ready and a 14.6 GiB memory.peak while serving on v4.4. An x86-64 CPU with AVX2+FMA+F16C and >= 40 logical CPUs: nv2 pins its threads to the measured layout (CPU lane 2-23, main thread 24, controller 25, NVMe readers 26-39) when the container's CPUs hold it, else derives the layout from the container's CPU set and logs it. Disk: the 125.3 GB checkpoint (only its non-expert weights are read) plus the 117.3 GB packed expert store on a local NVMe filesystem that supports O_DIRECT (xfs, ext4). This launch is NVMe-bound: ~82 NVMe records per token at C1 (vs ~22 at 55 GB), read at 19.9 GB/s during C1 decode (76% of a 4x Samsung 9100 PRO md RAID0 xfs array, ~170k IOPS); a single drive or a slower array decodes proportionally slower. PREPARE (once, CPU only): the image's `pack-store` packs the routed experts (42 layers x 288 x 9.44 MB) from the read-only checkpoint into /nvx/glm53_flash_exl3_3.05bpw_experts.bin + .json, re-reads every record with O_DIRECT and checks its sha256, and byte-compares a sample against the checkpoint; `verify-store` repeats the sha256 re-read. Serving mounts /nvx read-only. MEASURED (image smoke of v4.2-nvme, the same serving code; v4.3-nvme smoke: 8k prefill 650, C1 13.03; HOM-272 N129, EPYC 7443P, PCIe 4.0 x16, natural completions, NVMe reads uncapped): 8k prefill 654 tok/s, decode C1 12.72 tok/s; the same digest at a 55 GiB cap: 8k prefill 662, C1 17.25. QUALITY: same kernels as the 55 GB launch: prefill / teacher-forced panel exact (top-1 1.0000, KL 0); decode not exact (AVX2 CPU lane on RAM-resident picks; paired decode KL 0.005 measured at 55 GB). Text only; OpenAI tool calls parsed from GLM's <tool_call> format. IMAGE v4.6-nvme (glm53-flash-offload 0fbc0c5): GLM53_MODE=nvme now defaults to the N135 CPU-lane forward and the N136 decode switches (each settable to 0). MEASURED on this digest at 16 GiB (HOM-272 N129, same session as a v4.4 control at 12.89): 8k prefill 650, decode C1 14.38 tok/s.
# Full instructions: https://github.com/sybil-solutions/glm53-flash-offload/tree/0fbc0c569772863f9ae02730c3a3549a2b3b25f6

2 Download the weights

hf download turboderp/GLM-5.3-Flash-exl3 \
  --revision 332ab457b709b7ba30dd9a448be5de03b80a7ac9 \
  --local-dir ~/models/GLM-5.3-Flash-exl3-332ab457

3 Prepare the model (first start)

mkdir -p ~/models/glm-5.3-flash.exllamav3-nvme16g.128k-prepared-599016b56e69
docker run --rm \
  --user "$(id -u):$(id -g)" \
  -e HOME=/tmp \
  --memory 17179869184 \
  --memory-swap 17179869184 \
  --ulimit memlock=-1:-1 \
  --shm-size 1g \
  -e GLM53_EC_MAX_SLOTS=1376 \
  -e GLM53_MODE=nvme \
  -e GLM53_MODEL_DIR=/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw \
  -e GLM53_MODEL_DOWNLOAD=0 \
  -e GLM53_NV_STORE_DIR=/nvx \
  -e HF_HUB_OFFLINE=1 \
  --entrypoint /opt/glm53/docker/entrypoint.sh \
  -v ~/models/glm-5.3-flash.exllamav3-nvme16g.128k-prepared-599016b56e69:/nvx \
  -v ~/models/GLM-5.3-Flash-exl3-332ab457:/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw:ro \
  ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69 \
  pack-store

4 Verify the prepared model

docker run --rm --user "$(id -u):$(id -g)" -e HOME=/tmp --memory 17179869184 --memory-swap 17179869184 --ulimit memlock=-1:-1 --shm-size 1g -e GLM53_EC_MAX_SLOTS=1376 -e GLM53_MODE=nvme -e GLM53_MODEL_DIR=/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw -e GLM53_MODEL_DOWNLOAD=0 -e GLM53_NV_STORE_DIR=/nvx -e HF_HUB_OFFLINE=1 --entrypoint /opt/glm53/docker/entrypoint.sh -v ~/models/glm-5.3-flash.exllamav3-nvme16g.128k-prepared-599016b56e69:/nvx:ro -v ~/models/GLM-5.3-Flash-exl3-332ab457:/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw:ro ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69 verify-store

5 Start the server

docker run --rm \
  --gpus all \
  -p 8000:30000 \
  --memory 17179869184 \
  --memory-swap 17179869184 \
  --ulimit memlock=-1:-1 \
  --shm-size 1g \
  -e GLM53_EC_MAX_SLOTS=1376 \
  -e GLM53_MODE=nvme \
  -e GLM53_MODEL_DIR=/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw \
  -e GLM53_MODEL_DOWNLOAD=0 \
  -e GLM53_NV_STORE_DIR=/nvx \
  -e HF_HUB_OFFLINE=1 \
  -v ~/models/glm-5.3-flash.exllamav3-nvme16g.128k-prepared-599016b56e69:/nvx:ro \
  -v ~/models/GLM-5.3-Flash-exl3-332ab457:/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw:ro \
  --entrypoint /opt/glm53/docker/entrypoint.sh \
  ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69 \
  -m /models/turboderp-GLM-5.3-Flash-exl3-3.05bpw \
  -cs 131072 \
  --max-batch-size 8 \
  -chunk_size 8192 \
  -ambs 4 \
  --host 0.0.0.0 \
  --port 30000 \
  --served-name glm-5.3-flash

What it passed

✓ loadLoads and serves within an hour

✓ chatAnswers a plain question and stops on its own

✓ reasoningThinks separately and gets 17 × 23 right

✓ toolsCalls a tool with the right arguments and uses the result

✓ contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by owner in lab/runs/rtx-3090-24gb.glm-5.3-flash.exllamav3-glm-5.3-flash-exl3-3.05bpw-nvme-16g-128k-rtx-3090-24gb.128k.20261009T124547.json, where it reports 14.7 tok/s. Our six checks have not run on it yet.

Details

Weights: turboderp/GLM-5.3-Flash-exl3 @ 332ab457b7 ›
Image: ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69
Source: sybil-solutions/glm53-flash-offload @ 0fbc0c569772863f9ae027 ›
Launch: exllamav3-glm-5.3-flash-exl3-3.05bpw-nvme-16g-128k-rtx-3090-24gb ›
Recipe file: registry/recipes/nvidia/rtx-3090-24gb/glm-5.3-flash.exllamav3-nvme16g.128k.json ›
Model released: Aug 25, 2026
Tested: not yet by us; reported by owner