Nemotron 3.5 Bolt 74B-A9B

· llama.cpp · 256K context · Arc Pro B70 · 32 GB

NVIDIA's hybrid Mamba-Transformer MoE (74B total, about 9B active), compressed from Nemotron 3.5 Super VL and quantization-aware distilled to a 24 GB checkpoint (IQ2_XXS experts, NVFP4 Mamba layers, Q8_0 attention), with an image projector.

26 tok/sdecode

603 tok/sprefill

256Kcontext window

Tested on this cardOct 17, 2026

Run it

Follow the steps below to download and run this recipe. Omarchy Local AI lists the recipes it can launch on your hardware. The server then answers on http://localhost:8000/v1.

1 Set up first

# Requires one Intel Arc Pro B70 32 GB and 25 GB of disk for the GGUF and its image projector; the whole model sits in VRAM, so host RAM barely matters (the GGUF is memory-mapped). Upstream llama.cpp server-intel image build 11681 (commit 5e03a1ff), the first SYCL build with Nemotron 3.5 Bolt support (llama.cpp cd0a5b6a). Context 262,144 tokens in one slot (--parallel 1: more slots split the context). The LiLiCorr speculative drafter is not supported by llama.cpp (0 of 2,877 drafts accepted as a draft-simple model) and n-gram speculation gave no gain on the protocol's prompts, so neither is used. Measured on omarchy (EPYC 7443P, PCIe 4.0 x16) with the local-ai-recipe-kit sweep (--api llamacpp): prefill 731 / 736 / 728 / 707 tok/s at 8K / 16K / 32K / 64K; decode 26.2 / 26.1 / 26.1 / 26.1 tok/s at 1-4 users (one slot), C1 at 32K 24.6; needle 242,176 tokens at depth 0.8 pass. Vision: images through the NVIDIA mmproj.

2 Download the weights

hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  --revision e0fecb69cb3e5e4c5e8a056fa2aec98c78e91246 \
  --local-dir ~/models/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF-e0fecb69

3 Start the server

docker run --rm \
  --device /dev/dri \
  -p 8000:8080 \
  --shm-size 8g \
  -e ONEAPI_DEVICE_SELECTOR=level_zero:0 \
  -e UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1 \
  -v ~/models/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF-e0fecb69:/models:ro \
  ghcr.io/ggml-org/llama.cpp@sha256:b815baa10b515c60fb7400ebcd7e3b39ba7ae860df9ded0f6f7f4e52fee517fc \
  -m /models/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  --mmproj /models/mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  --alias nemotron-3.5-bolt \
  --host 0.0.0.0 \
  --port 8080 \
  --ctx-size 262144 \
  --n-gpu-layers 999 \
  --parallel 1 \
  --flash-attn on \
  --batch-size 2048 \
  --ubatch-size 2048 \
  --jinja \
  --reasoning-format deepseek \
  --metrics

What it passed

✓ loadLoads and serves within an hour

✓ chatAnswers a plain question and stops on its own

✓ reasoningThinks separately and gets 17 × 23 right

✓ toolsCalls a tool with the right arguments and uses the result

✓ contextRecalls a code buried in a prompt that fills 85% of the window

✓ speedDecodes at 15 tok/s or more

Passed all six checks on a real Intel Arc Pro B70 32GB on Oct 17, 2026.

Details

Weights: nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF @ e0fecb69cb ›
Image: ghcr.io/ggml-org/llama.cpp@sha256:b815baa10b515c60fb7400ebcd7e3b39ba7ae860df9ded0f6f7f4e52fee517fc
Launch: llama.cpp-sycl-nemotron-3.5-bolt-74b-a9b-24gb-256k-arc-pro-b70 ›
Recipe file: registry/recipes/intel/intel-arc-pro-b70-32gb/nemotron-3.5-bolt-74b-a9b.llama.cpp.256k.json ›
Model released: Oct 15, 2026
Tested: owner, Intel Arc Pro B70 32GB