DeepSeek-V4.1-Flash

EXL3 3 bpw · vllm · 256K context · RTX 3090 · 24 GB

DeepSeek's V4.1 Flash, a large mixture-of-experts model.

19 tok/sdecode

724 tok/sprefill

256Kcontext window

Reported by ownerOct 2, 2026

Run it

This is the launch as owner publishes it. Their repository has the full setup: https://github.com/sybil-solutions/dsv41-flash-offload › The server then answers on http://localhost:8000/v1.

1 Set up first

# NEEDS (measured on the running D119 server, README 'Host requirements'): 1x 24 GB sm_86 GPU (RTX 3090; 21,852 MiB in use at ready, the VRAM expert mirror cache takes what is left minus 900 MB; nothing else may use the GPU); host RAM: container anonymous 194.0 GiB (pinned experts 190.5 GiB + process), container peak incl. Engram page cache 212.1 GiB / 227.8 GB -> a 256 GiB host (30 GB buffer); an x86-64 CPU with AVX2+FMA+F16C and >= 24 cores (CPU expert tier pinned to cores 2-23, 22 threads: DSV41_CT_CPUS / DSV41_CT_THREADS); NVMe: 457.9 GB of inputs + image (Mia EXL3 3.0bpw 219.3 GB, Engram shards 47-48 of deepseek-ai/DeepSeek-V4.1-Flash 203.1 GB read on demand while serving, overlay pack 2.9 GB, image 32.6 GB) -> 488 GB free (1 TB drive); resources: --cap-add IPC_LOCK --ulimit memlock=-1 (typed: ipcLock, memlockUnlimited). ONE-TIME PACK (typed prepare): a writable prepared directory is mounted at /models with both weight dirs mounted read-only on top of it at their own paths, and the image runs with the single argument `prepare` (GPU, a few minutes): it builds /models/DSV41-EXL3-3090-D010 (symlinks into the two weight dirs, FP16 wo_a and compressor/indexer projections, text-only V4.1 config, host-plan.json) and checks it against the measured pack's sha256; `verify-pack` (CPU only) re-checks it; serving mounts the prepared /models read-only. IMAGE: published and pinned above, built through local-ai-images main with provenance. MEASURED (campaign run D119 = this launch: --max-model-len 262144, --kv-cache-memory 1.5 GiB = 297,224 fp8 KV tokens (1 GiB is refused at 262144), 16k chunks, staged-DMA prefill + FP16 GEMM, CPU tier, elastic VRAM expert cache; EPYC 7443P, 503 GiB DDR4, PCIe 4.0 x16, 990 PRO; bench/ctx_scan.py: TTFT prefill + inter-token decode, C1, natural EOS, no max_tokens; captured 2026-10-02 12:48-13:25 CEST; https://github.com/0xSero/dsv41-flash-offload/blob/2e9da5ecdab37c232c2da887e25c33a99da738d3/results/D119/ctx_scan.json and https://github.com/0xSero/dsv41-flash-offload/blob/2e9da5ecdab37c232c2da887e25c33a99da738d3/results/D119/ctx_scan_261k.json): prompt 8,192: prefill 589.1 / decode 16.38 tok/s; 32,768: 774.6 / 19.42; 65,536: 813.2 / 20.51; 131,072: 791.4 / 20.91; 196,608: 781.1 / 20.48; 261,000: 764.1 / 20.59. Earlier steps (P2 sweep at --max-model-len 65536, results/<run>/sweep.json): D030 baseline prefill 93.4 / 95.8, decode C1 4.10; D107 (1,024-token chunks) C1 21.34 / C2 26.27 / C4 28.30 agg; D061 prefill 565.2 / 744.9; D117 (non-elastic cache) 575.2 / 765.5, C1 15.55. Concurrency (C2/C4) at 262144 not measured. QUALITY: prefill of the staged-DMA path vs the D026 full-vocabulary reference (12 prompts, 416 positions): top-1 0.9856, mean KL 0.0068 nats, median 2.8e-6 - fails the campaign's inherited guard (top-1 >= 0.988, KL <= 0.00103); EXL3_DMA_GEMM=0 is the slower segmented path (results/D061/fidelity-verdict.json). LAB ACCEPTANCE: all six gates (load, chat, reasoning, tools, context, speed) passed on the pinned D119 image in D122; proof sha256:83647e119a6d18dd, decode 18.5 tok/s and prefill 724 tok/s. This is separate from the quality comparison above.
# Full instructions: https://github.com/0xSero/dsv41-flash-offload/tree/2e9da5ecdab37c232c2da887e25c33a99da738d3

2 Download the weights

hf download Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-3.0bpw \
  --revision c5534b90602b4090980a1c3ded1eb3b4d99a38d5 \
  --local-dir ~/models/DeepSeek-V4.1-Flash-EXL3-3.0bpw-c5534b90

hf download deepseek-ai/DeepSeek-V4.1-Flash config.json model.safetensors.index.json inference/engram.py model-00047-of-00048.safetensors model-00048-of-00048.safetensors \
  --revision 2cba9e42aa026125f3ed06c6d98c1db82f7ca027 \
  --local-dir ~/models/DeepSeek-V4.1-Flash-2cba9e42

3 Prepare the model (first start)

mkdir -p ~/models/deepseek-v4.1-flash.vllm.256k-prepared-f5639cc52d99
docker run --rm \
  --gpus all \
  --user "$(id -u):$(id -g)" \
  -e HOME=/tmp \
  --ulimit memlock=-1:-1 \
  --cap-add IPC_LOCK \
  --shm-size 8g \
  -e DSV41_CPU_TIER=1 \
  -e DSV41_EC=1 \
  -e DSV41_EC_ELASTIC=1 \
  -e DSV41_EC_MARGIN_MB=900 \
  -e DSV41_KV_CACHE_BYTES=1610612736 \
  -e DSV41_MAX_MODEL_LEN=262144 \
  -e DSV41_MODEL_ROOT=/models \
  -e DSV41_PACK=/models/DSV41-EXL3-3090-D010 \
  -e EXL3_DMA_GEMM=1 \
  -e EXL3_HOST_DMA=1 \
  -e HF_HUB_OFFLINE=1 \
  -e VLLM_EXL3_FAT_THRESHOLD=32 \
  --entrypoint /opt/dsv41/docker/entrypoint.sh \
  -v ~/models/deepseek-v4.1-flash.vllm.256k-prepared-f5639cc52d99:/models \
  -v ~/models/DeepSeek-V4.1-Flash-EXL3-3.0bpw-c5534b90:/models/Mia-DeepSeek-V4.1-Flash-EXL3-3.0bpw:ro \
  -v ~/models/DeepSeek-V4.1-Flash-2cba9e42:/models/DeepSeek-V4.1-Flash-engram:ro \
  ghcr.io/0xsero/dsv41-flash-offload@sha256:964fb0a1e0f94fb307b757b39067e731fb9fa086b8e5c09d97eaf5639cc52d99 \
  prepare

4 Verify the prepared model

docker run --rm --user "$(id -u):$(id -g)" -e HOME=/tmp --ulimit memlock=-1:-1 --cap-add IPC_LOCK --shm-size 8g -e DSV41_CPU_TIER=1 -e DSV41_EC=1 -e DSV41_EC_ELASTIC=1 -e DSV41_EC_MARGIN_MB=900 -e DSV41_KV_CACHE_BYTES=1610612736 -e DSV41_MAX_MODEL_LEN=262144 -e DSV41_MODEL_ROOT=/models -e DSV41_PACK=/models/DSV41-EXL3-3090-D010 -e EXL3_DMA_GEMM=1 -e EXL3_HOST_DMA=1 -e HF_HUB_OFFLINE=1 -e VLLM_EXL3_FAT_THRESHOLD=32 --entrypoint /opt/dsv41/docker/entrypoint.sh -v ~/models/deepseek-v4.1-flash.vllm.256k-prepared-f5639cc52d99:/models:ro -v ~/models/DeepSeek-V4.1-Flash-EXL3-3.0bpw-c5534b90:/models/Mia-DeepSeek-V4.1-Flash-EXL3-3.0bpw:ro -v ~/models/DeepSeek-V4.1-Flash-2cba9e42:/models/DeepSeek-V4.1-Flash-engram:ro ghcr.io/0xsero/dsv41-flash-offload@sha256:964fb0a1e0f94fb307b757b39067e731fb9fa086b8e5c09d97eaf5639cc52d99 verify-pack

5 Start the server

docker run --rm \
  --gpus all \
  -p 8000:8000 \
  --ulimit memlock=-1:-1 \
  --cap-add IPC_LOCK \
  --shm-size 8g \
  -e DSV41_CPU_TIER=1 \
  -e DSV41_EC=1 \
  -e DSV41_EC_ELASTIC=1 \
  -e DSV41_EC_MARGIN_MB=900 \
  -e DSV41_KV_CACHE_BYTES=1610612736 \
  -e DSV41_MAX_MODEL_LEN=262144 \
  -e DSV41_MODEL_ROOT=/models \
  -e DSV41_PACK=/models/DSV41-EXL3-3090-D010 \
  -e EXL3_DMA_GEMM=1 \
  -e EXL3_HOST_DMA=1 \
  -e HF_HUB_OFFLINE=1 \
  -e VLLM_EXL3_FAT_THRESHOLD=32 \
  -v ~/models/deepseek-v4.1-flash.vllm.256k-prepared-f5639cc52d99:/models:ro \
  -v ~/models/DeepSeek-V4.1-Flash-EXL3-3.0bpw-c5534b90:/models/Mia-DeepSeek-V4.1-Flash-EXL3-3.0bpw:ro \
  -v ~/models/DeepSeek-V4.1-Flash-2cba9e42:/models/DeepSeek-V4.1-Flash-engram:ro \
  --entrypoint /opt/dsv41/docker/entrypoint.sh \
  ghcr.io/0xsero/dsv41-flash-offload@sha256:964fb0a1e0f94fb307b757b39067e731fb9fa086b8e5c09d97eaf5639cc52d99 \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name deepseek-v4.1-flash \
  --max-model-len 262144 \
  --kv-cache-memory 1610612736 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 4 \
  --enable-auto-tool-choice \
  --tool-call-parser deepseek_v41 \
  --reasoning-parser deepseek_v41

What it passed

✓ loadLoads and serves within an hour

✓ chatAnswers a plain question and stops on its own

✓ reasoningThinks separately and gets 17 × 23 right

✓ toolsCalls a tool with the right arguments and uses the result

✓ contextRecalls a code buried in a prompt that fills 85% of the window

✓ speedDecodes at 15 tok/s or more

Published by owner in https://github.com/sybil-solutions/dsv41-flash-offload, where it reports 18.5 tok/s. Our six checks have not run on it yet. measured on the same image with the raw flags --cap-add IPC_LOCK --ulimit memlock=-1, which the typed launch carries as resources; the typed launch awaits its own lab run

Details

Weights: Mia-AiLab/DeepSeek-V4.1-Flash-EXL3-3.0bpw @ c5534b9060 ›
Image: ghcr.io/0xsero/dsv41-flash-offload@sha256:964fb0a1e0f94fb307b757b39067e731fb9fa086b8e5c09d97eaf5639cc52d99
Source: 0xSero/dsv41-flash-offload @ 2e9da5ecdab37c232c2da887e25c33a ›
Launch: vllm-deepseek-v4.1-flash-exl3-3.0bpw-offload-256k-rtx-3090-24gb-typed ›
Recipe file: registry/recipes/nvidia/rtx-3090-24gb/deepseek-v4.1-flash.vllm.256k.json ›
Model released: Sep 10, 2026
Tested: not yet by us; reported by owner