GLM-5.3-Flash

EXL3 3.05 bpw · exllamav3-nvme22g-32t · 128K context · RTX 3090 · 24 GB

Z.ai's GLM-5.3-Flash, a large mixture-of-experts model for multi-GPU machines.

13 tok/sdecode

1,031 tok/sprefill

128Kcontext window

Reported by ownerOct 17, 2026

Run it

This is the launch as owner publishes it. Their repository has the full setup: lab/runs/rtx-3090-24gb.glm-5.3-flash.exllamav3-glm-5.3-flash-exl3-3.05bpw-nvme-22g-32t-128k-rtx-3090-24gb.128k.20261017T032612.json › The server then answers on http://localhost:8000/v1.

1 Set up first

# NEEDS: 1x RTX 3090 24 GB (expert cache capped at 1,376 slots by GLM53_EC_MAX_SLOTS so a desktop on the same GPU cannot run the CUDA graphs out of memory); the container is capped at 22 GiB of host RAM (resources: --memory 22g --memory-swap 22g, swap off inside the cap) and runs with --ulimit memlock=-1 (the RAM tier is cudaHostRegister'ed). Below a 24 GiB cap the image's entrypoint (v4.2-nvme) applies a small-host budget: -rcs 1 (exllamav3 host recurrent-state cache 1 GiB instead of 4), GLM53_NV_MARGIN_GB=1.5, GLM53_NV_PF_RING=128 prefill read-ahead slots, and a deeper NVMe reader pool (GLM53_NV_THREADS=48 x GLM53_NV_PIECE_KB=1024), each overridable; NVMe modes are refused only under 15 GiB. The RAM tier sizes itself from the cap (1,393-1,652 experts = 12.2-14.5 GiB on the measured box, fewer when the page cache already holds part of the cap at start). An x86-64 CPU with AVX2+FMA+F16C and >= 32 logical CPUs: nv2 is pinned to a 16-core / 32-thread layout (main thread 0, controller 1, CPU lane 2-15, NVMe readers 18-31, the lane's SMT siblings on the measured box). Disk: the 125.3 GB checkpoint (only its non-expert weights are read) plus the 117.3 GB packed expert store on a local NVMe filesystem that supports O_DIRECT (xfs, ext4). This launch is NVMe-bound (measured on a 3x NVMe md RAID0 ext4 array: 2x Samsung 990 PRO + WD_BLACK SN850); a single drive or a slower array decodes proportionally slower. PREPARE (once, CPU only): the image's `pack-store` packs the routed experts (42 layers x 288 x 9.44 MB) from the read-only checkpoint into /nvx/glm53_flash_exl3_3.05bpw_experts.bin + .json, re-reads every record with O_DIRECT and checks its sha256, and byte-compares a sample against the checkpoint; `verify-store` repeats the sha256 re-read. Serving mounts /nvx read-only. MEASURED (lab.py try --on endpoint; Ryzen 9 9950X3D 16c/32t, 32 GB DDR5, PCIe 4.0 x16 with a 4K desktop on the same GPU, natural completions, v4.6-nvme): prefill 1,009 tok/s, decode C1 12.2-12.5 tok/s with GLM53_LA=0 and 12.8 tok/s with the glm-flash-lite #18 fix and the lookahead on; nvme16g on the same box with GLM53_LA=0: decode C1 9.8 tok/s. QUALITY: same kernels as the 55 GB launch: prefill / teacher-forced panel exact (top-1 1.0000, KL 0); decode not exact (AVX2 CPU lane on RAM-resident picks; paired decode KL 0.005 measured at 55 GB). Text only; OpenAI tool calls parsed from GLM's <tool_call> format. IMAGE v4.6.1-nvme (glm-flash-lite 0bd10f3): v4.6-nvme plus the fix for a decode-lookahead stall after the first max_rq_tokens requeue (glm-flash-lite #18); GLM53_MODE=nvme defaults to the N135 CPU-lane forward and the N136 decode switches (each settable to 0).
# Full instructions: https://github.com/sybil-solutions/glm-flash-lite/tree/0bd10f371abdcdb8c3648453f9da834c6dd2c980

2 Download the weights

hf download turboderp/GLM-5.3-Flash-exl3 \
  --revision 332ab457b709b7ba30dd9a448be5de03b80a7ac9 \
  --local-dir ~/models/GLM-5.3-Flash-exl3-332ab457

3 Prepare the model (first start)

mkdir -p ~/models/glm-5.3-flash.exllamav3-nvme22g-32t.128k-prepared-7752bacba45e
docker run --rm \
  --user "$(id -u):$(id -g)" \
  -e HOME=/tmp \
  --memory 23622320128 \
  --memory-swap 23622320128 \
  --ulimit memlock=-1:-1 \
  --shm-size 1g \
  -e GLM53_EC_MAX_SLOTS=1376 \
  -e GLM53_MAIN_CPUS=0 \
  -e GLM53_MODE=nvme \
  -e GLM53_MODEL_DIR=/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw \
  -e GLM53_MODEL_DOWNLOAD=0 \
  -e GLM53_NV_CPU_CPUS=2-15 \
  -e GLM53_NV_CTL_CPU=1 \
  -e GLM53_NV_READER_CPUS=18-31 \
  -e GLM53_NV_STORE_DIR=/nvx \
  -e HF_HUB_OFFLINE=1 \
  --entrypoint /opt/glm53/docker/entrypoint.sh \
  -v ~/models/glm-5.3-flash.exllamav3-nvme22g-32t.128k-prepared-7752bacba45e:/nvx \
  -v ~/models/GLM-5.3-Flash-exl3-332ab457:/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw:ro \
  ghcr.io/sybil-solutions/glm53-flash-offload@sha256:6f6dc68521c37d486de9bd1adbecf88f30134dcb328a93eaed8d7752bacba45e \
  pack-store

4 Verify the prepared model

docker run --rm --user "$(id -u):$(id -g)" -e HOME=/tmp --memory 23622320128 --memory-swap 23622320128 --ulimit memlock=-1:-1 --shm-size 1g -e GLM53_EC_MAX_SLOTS=1376 -e GLM53_MAIN_CPUS=0 -e GLM53_MODE=nvme -e GLM53_MODEL_DIR=/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw -e GLM53_MODEL_DOWNLOAD=0 -e GLM53_NV_CPU_CPUS=2-15 -e GLM53_NV_CTL_CPU=1 -e GLM53_NV_READER_CPUS=18-31 -e GLM53_NV_STORE_DIR=/nvx -e HF_HUB_OFFLINE=1 --entrypoint /opt/glm53/docker/entrypoint.sh -v ~/models/glm-5.3-flash.exllamav3-nvme22g-32t.128k-prepared-7752bacba45e:/nvx:ro -v ~/models/GLM-5.3-Flash-exl3-332ab457:/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw:ro ghcr.io/sybil-solutions/glm53-flash-offload@sha256:6f6dc68521c37d486de9bd1adbecf88f30134dcb328a93eaed8d7752bacba45e verify-store

5 Start the server

docker run --rm \
  --gpus all \
  -p 8000:30000 \
  --memory 23622320128 \
  --memory-swap 23622320128 \
  --ulimit memlock=-1:-1 \
  --shm-size 1g \
  -e GLM53_EC_MAX_SLOTS=1376 \
  -e GLM53_MAIN_CPUS=0 \
  -e GLM53_MODE=nvme \
  -e GLM53_MODEL_DIR=/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw \
  -e GLM53_MODEL_DOWNLOAD=0 \
  -e GLM53_NV_CPU_CPUS=2-15 \
  -e GLM53_NV_CTL_CPU=1 \
  -e GLM53_NV_READER_CPUS=18-31 \
  -e GLM53_NV_STORE_DIR=/nvx \
  -e HF_HUB_OFFLINE=1 \
  -v ~/models/glm-5.3-flash.exllamav3-nvme22g-32t.128k-prepared-7752bacba45e:/nvx:ro \
  -v ~/models/GLM-5.3-Flash-exl3-332ab457:/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw:ro \
  --entrypoint /opt/glm53/docker/entrypoint.sh \
  ghcr.io/sybil-solutions/glm53-flash-offload@sha256:6f6dc68521c37d486de9bd1adbecf88f30134dcb328a93eaed8d7752bacba45e \
  -m /models/turboderp-GLM-5.3-Flash-exl3-3.05bpw \
  -cs 131072 \
  --max-batch-size 8 \
  -chunk_size 8192 \
  -ambs 4 \
  --host 0.0.0.0 \
  --port 30000 \
  --served-name glm-5.3-flash

What it passed

✓ loadLoads and serves within an hour

✓ chatAnswers a plain question and stops on its own

✓ reasoningThinks separately and gets 17 × 23 right

✓ toolsCalls a tool with the right arguments and uses the result

✓ contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by owner in lab/runs/rtx-3090-24gb.glm-5.3-flash.exllamav3-glm-5.3-flash-exl3-3.05bpw-nvme-22g-32t-128k-rtx-3090-24gb.128k.20261017T032612.json, where it reports 12.8 tok/s. Our six checks have not run on it yet.

Details

Weights: turboderp/GLM-5.3-Flash-exl3 @ 332ab457b7 ›
Image: ghcr.io/sybil-solutions/glm53-flash-offload@sha256:6f6dc68521c37d486de9bd1adbecf88f30134dcb328a93eaed8d7752bacba45e
Source: sybil-solutions/glm-flash-lite @ 0bd10f371abdcdb8c3648453f9d ›
Launch: exllamav3-glm-5.3-flash-exl3-3.05bpw-nvme-22g-32t-128k-rtx-3090-24gb ›
Recipe file: registry/recipes/nvidia/rtx-3090-24gb/glm-5.3-flash.exllamav3-nvme22g-32t.128k.json ›
Model released: Aug 25, 2026
Tested: not yet by us; reported by owner