Inkling-Small

NVFP4 · sglang · 1024K context · 2 × DGX Spark GB10 · 256 GB

Thinking Machines' Inkling Small.

34 tok/sdecode

– tok/sprefill

1024Kcontext window

Reported by MiaAI-LabAug 2, 2026

Run it

This is the launch as MiaAI-Lab publishes it. Their repository has the full setup: MiaAI-Lab/Inkling-Small-NVFP4-Dual-DGX-Sparks@ce6bfbe90d6e › The server then answers on http://localhost:8000/v1.

1 Set up first

# Two Sparks linked over CX7 RoCEv2 (10.0.22.1/.2). Shared ed25519 key head->worker; worker mounts head HF cache via SSHFS at /mnt/head-hf-cache (user_allow_other in /etc/fuse.conf) and binds it to /root/.cache/huggingface. Image is pulled on head, tagged local/sglang-inkling:gb10-kvquant, docker save/load to worker. Download thinkingmachines/Inkling-Small-NVFP4 and RadixArk/Inkling-Small-DSpark-Preview into ~/.cache/huggingface on head.
# Full instructions: https://github.com/MiaAI-Lab/Inkling-Small-NVFP4-Dual-DGX-Sparks/tree/ce6bfbe90d6e680e7b8172ebd1922fa148158e26

2 Download the weights

hf download thinkingmachines/Inkling-Small-NVFP4 \
  --revision b6a99534467840620d411e4cd4ad5819b2610d9c

3 Start the server on each of the 2 machines (NODE_RANK 0 to 1)

docker run --rm \
  --gpus all \
  -p 8000:8888 \
  --network host \
  --gpus all \
  --shm-size 16g \
  --ipc host \
  --privileged \
  --device /dev/infiniband \
  --cap-add IPC_LOCK \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --shm-size 16g \
  -e FLASHINFER_CUDA_ARCH_LIST=12.1a \
  -e GLOO_SOCKET_IFNAME="${CX7_IF}" \
  -e HF_HOME=/root/.cache/huggingface \
  -e HF_HUB_CACHE=/root/.cache/huggingface \
  -e HF_HUB_OFFLINE=1 \
  -e HUGGINGFACE_HUB_CACHE=/root/.cache/huggingface \
  -e INKLING_COMMIT_STEP_BIAS=1 \
  -e INKLING_DRAFT_CTX_CAP=65536 \
  -e INKLING_NOOP_CONV_COMMIT=0 \
  -e INKLING_TORCH_CONV_COMMIT=1 \
  -e NCCL_CROSS_NIC=0 \
  -e NCCL_CUMEM_ENABLE=0 \
  -e NCCL_DEBUG=WARN \
  -e NCCL_IB_DISABLE=0 \
  -e NCCL_IB_GID_INDEX=3 \
  -e NCCL_IB_HCA="${CX7_IB}" \
  -e NCCL_IB_ROCE_VERSION_NUM=2 \
  -e NCCL_IGNORE_CPU_AFFINITY=1 \
  -e NCCL_NET=IB \
  -e NCCL_NET_PLUGIN=none \
  -e NCCL_NVLS_ENABLE=0 \
  -e NCCL_SOCKET_IFNAME="${CX7_IF}" \
  -e PYTHONWARNINGS=ignore::SyntaxWarning \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
  -e TORCH_CUDA_ARCH_LIST=12.1a \
  -e TP_SOCKET_IFNAME="${CX7_IF}" \
  -e TRANSFORMERS_CACHE=/root/.cache/huggingface \
  -e TRANSFORMERS_OFFLINE=1 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint python3 \
  ghcr.io/drowzeys/inkling-sglang-gb10:kvquant@sha256:079775a8deb415d16b89a505df0795987add3395e6e63a42a885a712ccb1ddc3 \
  -m sglang.launch_server \
  --model-path "${CONTAINER_MODEL}" \
  --trust-remote-code \
  --served-model-name inkling-small \
  --tp-size 2 \
  --context-length 1048576 \
  --quantization modelopt_fp4 \
  --attention-backend triton \
  --page-size 1 \
  --fp4-gemm-backend flashinfer_trtllm \
  --moe-runner-backend marlin \
  --mamba-radix-cache-strategy extra_buffer \
  --mem-fraction-static 0.85 \
  --swa-full-tokens-ratio 0.1 \
  --mamba-full-memory-ratio 0.1 \
  --max-running-requests 16 \
  --chunked-prefill-size 8192 \
  --reasoning-parser inkling \
  --tool-call-parser inkling \
  --skip-server-warmup \
  --disable-flashinfer-autotune \
  --stream-interval 32 \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path "${CONTAINER_DRAFT}" \
  --speculative-draft-model-quantization unquant \
  --speculative-dspark-block-size 7 \
  --disable-prefill-cuda-graph \
  --disable-piecewise-cuda-graph \
  --triton-attention-reduce-in-fp32 \
  --kv-cache-dtype fp4_mx_block16 \
  --cuda-graph-bs 1 \
  2 \
  3 \
  4 \
  5 \
  6 \
  7 \
  8 \
  10 \
  12 \
  14 \
  16 \
  --nnodes 2 \
  --node-rank "${NODE_RANK}" \
  --dist-init-addr 10.0.22.1:25000 \
  --host 0.0.0.0 \
  --port 8888

What it passed

– loadLoads and serves within an hour

– chatAnswers a plain question and stops on its own

– reasoningThinks separately and gets 17 × 23 right

– toolsCalls a tool with the right arguments and uses the result

– contextRecalls a code buried in a prompt that fills 85% of the window

– speedDecodes at 15 tok/s or more

Published by MiaAI-Lab in MiaAI-Lab/Inkling-Small-NVFP4-Dual-DGX-Sparks@ce6bfbe90d6e, where it reports 33.9 tok/s. Our six checks have not run on it yet.

Details

Weights: thinkingmachines/Inkling-Small-NVFP4 @ b6a9953446 ›
Image: ghcr.io/drowzeys/inkling-sglang-gb10:kvquant@sha256:079775a8deb415d16b89a505df0795987add3395e6e63a42a885a712ccb1ddc3
Source: MiaAI-Lab/Inkling-Small-NVFP4-Dual-DGX-Sparks @ ce6bfbe90d6e ›
Launch: sglang-inkling-small-nvfp4-1024k-2x ›
Recipe file: registry/recipes/nvidia/dgx-spark-gb10-128gb/inkling-small.sglang.1024k.2x.json ›
Model released: Jul 27, 2026
Tested: not yet by us; reported by miaai-lab