GLM-5.3-Flash

Follow the steps below to download and run this recipe. Omarchy Local AI lists the recipes it can launch on your hardware. The server then answers on http://localhost:8000/v1.

1. 1 Set up firstCopy

   # NEEDS: 1x RTX 3090 24 GB (the expert cache takes all free VRAM after load: ~22.7 GiB at ready, nothing else may use the GPU); ~238 GB of free system RAM (measured MemAvailable drop to its lowest point 237.7 GB in the image smoke P001, 228.9-234.4 GB at ready in G067/C056: two 114.2 GB copies of all routed experts, one pinned for GPU zero-copy reads, one block-contiguous for the CPU tier); an x86-64 CPU with AVX2+FMA+F16C and ~24 physical cores (CPU tier threads auto-sized: one per physical core minus 2); 125.3 GB of weights on a local NVMe SSD (read once at load, ~2 min from page cache). Measured on: EPYC 7443P 24C, 8-channel DDR4 503 GiB, PCIe 4.0 x16, Samsung 990 PRO, driver 610.57. The image entrypoint holds the defaults (GLM53_MODE=fast = campaign run G067: expert-cache reserve 1.5 GB, prefill staging 2 x 2.6 GB, elastic 10 GB, CPU tier, fused hyper-connection sites, fast tier split, side-stream shared expert, exact Triton autotune pins); the README run adds --ulimit memlock=-1, but this launch (no ulimit flag) also loaded and served on driver 610.57. Protocol sweep (P2): prefill 8k 710 / 32k 951 tok/s; decode aggregate C1 28.15 / C2 31.57 / C4 33.98 tok/s, C1 at 32k 26.89. Quality: teacher-forced panel vs stock exllamav3 top-1 1.0000 / KL 0 (prefill path); decode with the CPU tier is NOT exact: paired check mean KL 0.0053 (p99 0.086), top-1 0.977 vs the exact GPU-only path (fix in progress; this recipe stays candidate/reported until it lands). GLM53_MODE=exact (no CPU tier, ~119 GB RAM, bit-exact): prefill 697 / 945, decode ~12.6 tok/s. Text only, OpenAI tool calls parsed from GLM's <tool_call> format. Kit target: github.com/0xSero/local-ai-recipe-kit targets/glm-5.3-flash-offload.md. Image smoke P002b (this digest bb633b0b, repo fe97bcf, README command, bridge network; the host's one remaining RTX 3090, which also drives the display, ~1 GB less free VRAM): panel 1.0000 / KL 0; prefill 8k 700.2 / 32k 958.0; decode C1 28.88 / C2 31.27 / C4 35.00, C1 at 32k 27.56; ready after 201 s. Streaming delivery (client-side SSE timestamps, 10-s bins, full answer) after 512- and 32k-token prompts on /generate and chat: 28.65-28.98 tok/s per answer, bins 26.4-30.4, 0.9-1.4 % of the text in the last second (no end-of-answer flush). Lab gates on this launch: load chat reasoning tools context speed all pass (context prompt 93,437 tokens; speed 28.8 tok/s over the first 30 s, run right after the context gate; 330 s). The previous digest 1159044a (repo c4b9160) measured the same sweep (P001: 707 / 953, 29.03 / 31.38 / 34.88) but delivered streams at ~half the generation rate because the server awaited a client-disconnect check after every SSE event (lab speed gate 14.4 tok/s); fixed in fe97bcf (disconnect polled by a side task every 0.1 s). Raw: github.com/0xSero/glm53-flash-offload results/P002b-stream-fix-smoke. IMAGE v4.4-nvme (glm53-flash-offload 7294f60). Changes since fe97bcf that reach fast mode: the AVX2 CPU tier clamps its 16-byte trellis window to the 96-byte tile (an out-of-bounds read that could SIGSEGV at the end of a mapping; output bit-identical); usage.completion_tokens counts across max_rq_tokens requeues; the NVMe-mode additions in expert_cache.py, serve.py and the entrypoint are off in GLM53_MODE=fast.
   # Full instructions: https://github.com/sybil-solutions/glm53-flash-offload/tree/7294f60e8e73dffff2cfce3db6b68f4d6ff06d31

2. 2 Download the weightsCopy

   hf download turboderp/GLM-5.3-Flash-exl3 \
     --revision 332ab457b709b7ba30dd9a448be5de03b80a7ac9 \
     --local-dir ~/models/GLM-5.3-Flash-exl3-332ab457

3. 3 Start the serverCopy

   docker run --rm \
     --gpus all \
     -p 8000:30000 \
     --shm-size 16g \
     -e GLM53_MODE=fast \
     -e GLM53_MODEL_DIR=/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw \
     -e GLM53_MODEL_DOWNLOAD=0 \
     -e HF_HUB_OFFLINE=1 \
     -v ~/models/GLM-5.3-Flash-exl3-332ab457:/models/turboderp-GLM-5.3-Flash-exl3-3.05bpw:ro \
     --entrypoint /opt/glm53/docker/entrypoint.sh \
     ghcr.io/sybil-solutions/glm53-flash-offload@sha256:c5a58a40bd93e57227d3986b81e9648d422c465a3ff00e14f17dad65dadf2310 \
     -m /models/turboderp-GLM-5.3-Flash-exl3-3.05bpw \
     -cs 131072 \
     --max-batch-size 8 \
     -chunk_size 8192 \
     -ambs 4 \
     --host 0.0.0.0 \
     --port 30000 \
     --served-name glm-5.3-flash

Passed all six checks on a real RTX 3090 on Oct 9, 2026.

Weights

turboderp/GLM-5.3-Flash-exl3 @ 332ab457b7 ›

Image

ghcr.io/sybil-solutions/glm53-flash-offload@sha256:c5a58a40bd93e57227d3986b81e9648d422c465a3ff00e14f17dad65dadf2310

Source

sybil-solutions/glm53-flash-offload @ 7294f60e8e73dffff2cfce ›

Launch

exllamav3-glm-5.3-flash-exl3-3.05bpw-offload-128k-rtx-3090-24gb ›

Recipe file

registry/recipes/nvidia/rtx-3090-24gb/glm-5.3-flash.exllamav3.128k.json ›

Model released

Aug 25, 2026

Tested

owner, RTX 3090