Nemotron 3.5 Lightning 30B-A3B

NVFP4 · sglang · 1024K context · RTX 6000 Ada · 48 GB

NVIDIA's Nemotron 3.5 Lightning, a small fast mixture-of-experts.

– tok/sdecode

– tok/sprefill

1024Kcontext window

Reported by MiaAI-LabAug 11, 2026

Run it

This is the launch as MiaAI-Lab publishes it. Their repository has the full setup: MiaAI-Lab/Nemotron3.5-Lightning-DGX-Spark-RTX-5090-6000-PRO@38c16a403ea5 › The server then answers on http://localhost:8000/v1.

1 Set up first

# start.sh (or --download-only) downloads target + DSpark draft into $HF_HOME (~/.cache/huggingface) via hf/huggingface-cli or a docker fallback; container runs `sglang serve` with snapshot paths resolved from the HF cache.
# Full instructions: https://github.com/MiaAI-Lab/Nemotron3.5-Lightning-DGX-Spark-RTX-5090-6000-PRO/tree/38c16a403ea57f26f1828fa572270245ae51f01d

2 Download the weights

hf download nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --revision bee7596271d1495f6992ae224aefde4410e816b8

3 Start the server

docker run --rm \
  --gpus all \
  -p 8000:8888 \
  --network host \
  --ulimit memlock=-1:-1 \
  --cap-add=IPC_LOCK \
  --ipc host \
  --gpus all \
  -v ${HF_HOME}:/root/.cache/huggingface \
  -v ${WORK_DIR}:/workspace \
  --workdir /workspace \
  --shm-size 32g \
  -e HF_HOME=/root/.cache/huggingface \
  -e HF_TOKEN="${HF_TOKEN}" \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  lmsysorg/sglang:dev-nemotron3-5-lightning@sha256:a04d9a1a7ffe371b05230aecab001d4ba2bfa0e5c137bc56409ecc4cbc3ac864 \
  sglang \
  serve \
  --model-path "${MODEL_IN_CONTAINER}" \
  --mamba-ssm-dtype float16 \
  --mem-fraction-static "${MEM_FRACTION_STATIC}" \
  --cuda-graph-max-bs-decode 2 \
  --reasoning-parser nemotron_3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path "${DRAFT_IN_CONTAINER}" \
  --speculative-dspark-block-size 3 \
  --host 0.0.0.0 \
  --port 8888

What it passed

– loadLoads and serves within an hour

– chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by MiaAI-Lab in MiaAI-Lab/Nemotron3.5-Lightning-DGX-Spark-RTX-5090-6000-PRO@38c16a403ea5. Our six checks have not run on it yet.

Details

Weights: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 @ bee7596271 ›
Image: lmsysorg/sglang:dev-nemotron3-5-lightning@sha256:a04d9a1a7ffe371b05230aecab001d4ba2bfa0e5c137bc56409ecc4cbc3ac864
Source: MiaAI-Lab/Nemotron3.5-Lightning-DGX-Spark-RTX-5090-6000-PRO ›
Launch: sglang-nemotron-3.5-lightning-30b-a3b-nvfp4-1024k-rtx-6000-ada-48gb ›
Recipe file: registry/recipes/nvidia/rtx-6000-ada-48gb/nemotron-3.5-lightning-30b-a3b.sglang.1024k.json ›
Model released: Aug 4, 2026
Tested: not yet by us; reported by miaai-lab