GLM-5.3-Flash

EXL3 3.05 bpw · exllamav3-nvme · 128K context · RTX 3090 · 24 GB

Z.ai's GLM-5.3-Flash, a large mixture-of-experts model for multi-GPU machines.

20 tok/sdecode

1,015 tok/sprefill

128Kcontext window

Tested on this cardOct 9, 2026

Run it

Follow the steps below to download and run this recipe. Omarchy Local AI lists the recipes it can launch on your hardware. The server then answers on http://localhost:8000/v1.

1 Set up first

# NEEDS: 1x RTX 3090 24 GB (expert cache capped at 1,376 slots by GLM53_EC_MAX_SLOTS so a desktop on the same GPU cannot run the CUDA graphs out of memory); the container is capped at 55 GiB of host RAM (resources: --memory 55g --memory-swap 55g, swap off inside the cap; the RAM tier is sized from the cap: 4,831 experts = 42.5 GiB; memory.peak 51.1-51.3 GiB measured) and runs with --ulimit memlock=-1 (the RAM tier is cudaHostRegister'ed); an x86-64 CPU with AVX2+FMA+F16C and >= 40 logical CPUs: nv2 pins its threads to the measured layout (CPU lane 2-23, main thread 24, controller 25, 16 NVMe readers 26-39) when the container's CPUs hold it, else derives the layout from the container's CPU set and logs it (speed with a derived layout is not measured); disk: the 125.3 GB checkpoint (only its non-expert weights are read) plus the 117.3 GB packed expert store on a local NVMe filesystem that supports O_DIRECT (xfs, ext4), read at ~7 GB/s on average while serving (20.2 TB over the 46-minute S3b2 sweep; measured on 4x Samsung 9100 PRO md RAID0 xfs; one drive is not measured). PREPARE (once, CPU only): the image's `pack-store` packs the routed experts (42 layers x 288 x 9.44 MB) from the read-only checkpoint into /nvx/glm53_flash_exl3_3.05bpw_experts.bin + .json, re-reads every record with O_DIRECT and checks its sha256, and byte-compares a sample against the checkpoint; `verify-store` repeats the sha256 re-read (11 s on the measured array). pack-store, verify-store and the server first probe O_DIRECT on /nvx and stop with a one-line reason if the filesystem refuses it. Serving mounts /nvx read-only. MEASURED (README '55 GB NVMe mode', campaign arm S3b2, sweep.py --template glm --prefill 8192 32768 --conc 1 2 4, natural completions, EPYC 7443P, 503 GiB DDR4, PCIe 4.0 x16, driver 610.57): prefill 8k 583 / 32k 812 tok/s; decode C1 14.85, C2 15.62 agg, C4 16.27 agg, C1 at 32k 13.46 tok/s. Image smoke P003 of digest 82eef823 (repo 9269860, the same serving code) on a busier host (desktop stream on the GPU, a B70 job running): 545.3 / 767.4, decode 13.70 / 14.38 / 15.24, 32k C1 12.40; ready after 76 s. QUALITY: prefill / teacher-forced panel top-1 1.0000, KL 0 (2,154 positions); decode not exact (CPU lane): paired KL 0.0047 (p99 0.082), top-1 0.986; GLM53_MODE=nvme-exact is bit-exact at ~8.3 tok/s. Text only; OpenAI tool calls parsed from GLM's <tool_call> format. Raw: github.com/0xSero/glm53-flash-offload results/N119-S3b2-55g and results/P003-nvme-image-smoke. IMAGE v4.6-nvme (glm53-flash-offload 0fbc0c5): GLM53_MODE=nvme now defaults to the N135 CPU-lane forward (GLM53_NV_CPU_KERN=1) and the N136 decode switches (GLM53_LA decode lookahead, GLM53_NV_PUBFAST, GLM53_NV_PFSIDE, GLM53_LA_BTTRIM), each settable to 0; also carries the v4.3 CPU-kernel window clamp and the v4.4 usage-count fix. MEASURED on this digest (HOM-272 N129, same session as a v4.4 control at 17.32 tok/s C1): 8k prefill 664, decode C1 19.47 / C2 19.09 / C4 19.75 agg; 32k prefill 969, C1 18.30 / C2 18.44. Paired decode KL 0.0077 (top-1 0.977).
# Full instructions: https://github.com/sybil-solutions/glm53-flash-offload/tree/0fbc0c569772863f9ae02730c3a3549a2b3b25f6

2 Download the weights

hf download turboderp/GLM-5.3-Flash-exl3 \
  --revision 332ab457b709b7ba30dd9a448be5de03b80a7ac9 \
  --local-dir ~/models/GLM-5.3-Flash-exl3-332ab457

3 Prepare the model (first start)

mkdir -p ~/models/glm-5.3-flash.exllamav3-nvme.128k-prepared-599016b56e69
docker run --rm \
  --user "$(id -u):$(id -g)" \
  -e HOME=/tmp \
  --memory 59055800320 \
  --memory-swap 59055800320 \
  --ulimit memlock=-1:-1 \
  --shm-size 1g \
  -e GLM53_EC_MAX_SLOTS=1376 \
  -e GLM53_MODE=nvme \
  -e GLM53_MODEL_DIR=/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw \
  -e GLM53_MODEL_DOWNLOAD=0 \
  -e GLM53_NV_STORE_DIR=/nvx \
  -e HF_HUB_OFFLINE=1 \
  --entrypoint /opt/glm53/docker/entrypoint.sh \
  -v ~/models/glm-5.3-flash.exllamav3-nvme.128k-prepared-599016b56e69:/nvx \
  -v ~/models/GLM-5.3-Flash-exl3-332ab457:/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw:ro \
  ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69 \
  pack-store

4 Verify the prepared model

docker run --rm --user "$(id -u):$(id -g)" -e HOME=/tmp --memory 59055800320 --memory-swap 59055800320 --ulimit memlock=-1:-1 --shm-size 1g -e GLM53_EC_MAX_SLOTS=1376 -e GLM53_MODE=nvme -e GLM53_MODEL_DIR=/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw -e GLM53_MODEL_DOWNLOAD=0 -e GLM53_NV_STORE_DIR=/nvx -e HF_HUB_OFFLINE=1 --entrypoint /opt/glm53/docker/entrypoint.sh -v ~/models/glm-5.3-flash.exllamav3-nvme.128k-prepared-599016b56e69:/nvx:ro -v ~/models/GLM-5.3-Flash-exl3-332ab457:/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw:ro ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69 verify-store

5 Start the server

docker run --rm \
  --gpus all \
  -p 8000:30000 \
  --memory 59055800320 \
  --memory-swap 59055800320 \
  --ulimit memlock=-1:-1 \
  --shm-size 1g \
  -e GLM53_EC_MAX_SLOTS=1376 \
  -e GLM53_MODE=nvme \
  -e GLM53_MODEL_DIR=/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw \
  -e GLM53_MODEL_DOWNLOAD=0 \
  -e GLM53_NV_STORE_DIR=/nvx \
  -e HF_HUB_OFFLINE=1 \
  -v ~/models/glm-5.3-flash.exllamav3-nvme.128k-prepared-599016b56e69:/nvx:ro \
  -v ~/models/GLM-5.3-Flash-exl3-332ab457:/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw:ro \
  --entrypoint /opt/glm53/docker/entrypoint.sh \
  ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69 \
  -m /models/turboderp-GLM-5.3-Flash-exl3-3.05bpw \
  -cs 131072 \
  --max-batch-size 8 \
  -chunk_size 8192 \
  -ambs 4 \
  --host 0.0.0.0 \
  --port 30000 \
  --served-name glm-5.3-flash

What it passed

✓ loadLoads and serves within an hour

✓ chatAnswers a plain question and stops on its own

✓ reasoningThinks separately and gets 17 × 23 right

✓ toolsCalls a tool with the right arguments and uses the result

✓ contextRecalls a code buried in a prompt that fills 85% of the window

✓ speedDecodes at 15 tok/s or more

Passed all six checks on a real RTX 3090 on Oct 9, 2026.

Details

Weights: turboderp/GLM-5.3-Flash-exl3 @ 332ab457b7 ›
Image: ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69
Source: sybil-solutions/glm53-flash-offload @ 0fbc0c569772863f9ae027 ›
Launch: exllamav3-glm-5.3-flash-exl3-3.05bpw-nvme-55g-128k-rtx-3090-24gb ›
Recipe file: registry/recipes/nvidia/rtx-3090-24gb/glm-5.3-flash.exllamav3-nvme.128k.json ›
Model released: Aug 25, 2026
Tested: owner, RTX 3090