GLM-5.1: SGLang Heterogeneous Deployment and Interleaved Thinking Configuration

One-sentence takeaway

Local deployment of GLM-5.1 depends on the exact transformers==5.3.0 version and SGLang parser configuration; coding Agent workflows must enable Interleaved + Preserved Thinking mode to prevent multi-turn forgetting.

Use cases

• Suitable tasks: Local private deployment, high-concurrency API proxies, and long-horizon task execution by Coding Agents (Claude Code, Roo Code, Kilo Code).

• Unsuitable tasks: Sharing the same Python virtual environment with legacy models that depend on transformers==4.x (such as older versions of DeepSeek).

• Applicable model versions: GLM-5.1, GLM-5.1-FP8.

• Applicable clients, Agents, or APIs: SGLang (v0.5.10+), KT-Kernel, vLLM, Claude Code, Roo Code, Kilo Code.

• Recommended reasoning levels and parameters:

  • General conversation: temperature=1.0, top_p=0.95, max_tokens=131072.

  • Terminal coding tasks (Terminal Bench): temperature=0.7, top_p=1.0, max_tokens=16384, context_length=202752.

  • Multi-turn tool calls (Tau2-Bench): temperature=0, max_tokens=16384, with preserved thinking enabled.

Ready-to-use content

SGLang + KT-Kernel heterogeneous inference launch command (FP8 precision)

export PYTORCH_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_JIT_DEEPGEMM=0

python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 30000 \
  --model /path/to/GLM-5.1-FP8 \
  --kt-weight-path /path/to/GLM-5.1-FP8 \
  --kt-cpuinfer 96 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 30 \
  --kt-method FP8 \
  --kt-gpu-prefill-token-threshold 1024 \
  --kt-enable-dynamic-expert-update \
  --kt-expert-placement-strategy uniform \
  --trust-remote-code \
  --mem-fraction-static 0.75 \
  --served-model-name GLM5.1 \
  --enable-mixed-chunk \
  --tensor-parallel-size 8 \
  --enable-p2p-check \
  --disable-shared-experts-fusion \
  --chunked-prefill-size 16384 \
  --max-running-requests 4 \
  --max-total-tokens 128000 \
  --attention-backend flashinfer \
  --fp8-gemm-backend cutlass \
  --kv-cache-dtype bf16 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --watchdog-timeout 3000

Agent workflow Thinking Mode request-body configuration

For multi-turn Coding Agents (such as Claude Code / Roo Code / API proxies), multi-turn thinking history must be preserved:

{
  "model": "GLM5.1",
  "messages": [
    {
      "role": "user",
      "content": "Analyze the codebase and implement the missing test cases."
    }
  ],
  "temperature": 0.7,
  "top_p": 1.0,
  "max_tokens": 16384,
  "chat_template_kwargs": {
    "enable_thinking": true,
    "clear_thinking": false
  }
}

To disable thinking mode completely (for ordinary low-latency completion only):

{
  "model": "GLM5.1",
  "messages": [
    {
      "role": "user",
      "content": "Summarize this diff."
    }
  ],
  "chat_template_kwargs": {
    "enable_thinking": false
  }
}

Test/workflow steps

1. Create an independent Python virtual environment and force-install the specified dependencies: pip install transformers==5.3.0 sglang-kt.

2. Ensure that the CUDA version is ≥ 12.0 (CUDA 12.8+ is recommended for FP8), and compile the dedicated acceleration operators through kt-kernel/install.sh.

3. Start the SGLang service. You must explicitly pass --tool-call-parser glm47 and --reasoning-parser glm45; otherwise, tool-call and chain-of-thought parsing may fail intermittently.

4. Set the base URL to http://localhost:30000/v1 in the OpenAI-compatible client, and inject chat_template_kwargs into the request.

5. For OOM issues, first fine-tune --kt-num-gpu-experts and --chunked-prefill-size.

Original evidence and data

• The official documentation clearly states that GLM-5.1 enables thinking mode by default and supports Interleaved Thinking (for regular interactions) and Interleaved + Preserved Thinking (recommended for Coding Agents).

• In long, multi-turn tasks (such as Tau2-Bench and Terminal-Bench 2), not preserving thinking causes breaks in the model's multi-turn strategy.

• Dependency constraint: transformers==5.3.0 is a hard requirement; the 4.x version installed by a standard pip setup cannot load the model.

Applicability boundaries

• CPU-GPU heterogeneous inference places high demands on system memory bandwidth and PCI-E lanes; offloading too many experts to the CPU can significantly increase time to first token (TTFT).

• transformers==5.3.0 conflicts with some legacy architecture models (such as older versions of DeepSeek) and should not be installed in the same environment.

• Preserved thinking mode consumes more of the context window as the number of turns increases, so max-total-tokens must be set appropriately.

Source excerpt or observation (for compliant short quote only)

The official tutorial explicitly marks Interleaved + Preserved Thinking as “Recommended for agentic workflows, especially code agents (e.g., Claude Code, Roo Code, Kilo Code)” and identifies it as the core switch for maintaining coherent reasoning in long-horizon tasks.