Kimi K3 vLLM Multi-Node Inference Configuration

One-sentence takeaway

This official recipe provides Kimi K3’s vLLM container, GPU/driver requirements, cross-node communication backends, and an OpenAI-compatible client example, making it suitable for an initial deployment feasibility check.

Use cases

• Suitable tasks: Data-center-grade self-hosting, visual-input serving, long-context inference, and multi-node MoE deployment.

• Unsuitable tasks: Personal computers, a single consumer GPU, or environments without CUDA 13/r580+ driver requirements.

• Applicable model version: moonshotai/Kimi-K3; the page still carries a pre-release label, so follow the current container and recipe state.

• Applicable client, agent, or API: vLLM OpenAI-compatible server, Python OpenAI SDK.

• Recommended reasoning tier and parameters: The page does not fix reasoning effort; the example uses max_tokens=2048, while max-model-len should be adjusted for the task in long-context scenarios.

Ready-to-use content

Image and hardware baseline:

vllm/vllm-openai:kimi-k3
CUDA 13 (cu130); host requires r580+ NVIDIA driver
NVIDIA: at least 8× GB300; multi-node for production traffic
ROCm: vllm/vllm-openai_rocm:kimi-k3; at least 8× MI355X/MI350X

OpenAI-compatible client and visual input:

import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    timeout=3600,
)

messages = [{
    "role": "user",
    "content": [
        {
            "type": "image_url",
            "image_url": {
                "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
            },
        },
        {"type": "text", "text": "Read all the text in the image."},
    ],
}]

start = time.time()
response = client.chat.completions.create(
    model="moonshotai/Kimi-K3",
    messages=messages,
    max_tokens=2048,
)
print(f"Response costs: {time.time() - start:.2f}s")
print(response.choices[0].message.content)

Test/workflow steps

1. On nodes with at least 8 GB300 or MI355X/MI350X cards, select the corresponding container and first verify driver, CUDA/ROCm, and available VRAM.

2. Start the vLLM OpenAI-compatible server. For cross-node RDMA, use --all2all-backend deepep_v2; for NVLink, use --all2all-backend flashinfer_nvlink_one_sided.

3. For MoE deployment, try deep_gemm_mega_moe; the recipe notes that this backend is incompatible with cross-node RDMA, so choose according to the interconnect topology.

4. Use the visual request above as a smoke test, then test pure text, tool calls, and different max-model-len values.

5. If enabling Model Runner v2/Rust Frontend, record the versions and results for VLLM_USE_V2_MODEL_RUNNER=1 and VLLM_USE_RUST_FRONTEND=1.

6. Tool calls must undergo schema validation. The page warns that K3 occasionally generates a format its own parser does not accept, which requires validation and retry.

Original evidence and data

• The vLLM recipe explicitly gives vllm/vllm-openai:kimi-k3, CUDA 13/cu130, an r580+ driver, and a minimum of 8× GB300 cards.

• The AMD recipe uses vllm/vllm-openai_rocm:kimi-k3, with a minimum of 8× MI355X/MI350X cards.

• The page lists specific notes on RDMA, NVLink, MoE backends, FP8 KV, DCP, NCCL, and the tool-call parser.

• The example uses localhost:8000/v1, model name moonshotai/Kimi-K3, timeout=3600, and max_tokens=2048.

Scope and limitations

• The recipe is a deployment guide, not a throughput, latency, or quality benchmark; “at least 8 cards” cannot be used to infer production capacity or available concurrency at 1M context.

• The page is still marked pre-release, and the container, driver, backend, and parameters may change as vLLM/Kimi K3 updates.

• Do not treat the example image URL as business data; replace it with compliant test samples owned by you in production.

Reproduction steps

Record the vLLM image digest, driver/CUDA, GPU model, and interconnect. Following the recipe, run three smoke-test groups: visual OCR, long text, and tool calls. For each group, record time to first token, generation speed, VRAM, KV cache, parser retries, and error logs; then change max-model-len to build a capacity curve.

Source excerpt or observation (brief excerpt for compliance only)

The original hardware prerequisite is: “At least 8x GB300.”