r/LocalLLaMA Field Test: Running GLM 5.2 (744B MoE) Locally Without a GPU with the Colibrì Engine

Core content summary

u/Lopsided_Dot_4557 shared a hands-on test of running GLM 5.2 (744B MoE) on a machine without a GPU, using colibrì, a pure-C engine that streams MoE experts from disk:

Principle and configuration

• A 744B MoE activates only a small number of experts for each token: colibrì keeps the dense portion (about 10GB) resident in RAM and streams the experts selected by the router from disk as needed.

• The complete int4 model occupies about 370GB on disk; it does not need to fit entirely in memory.

• Test machine: a single machine with 132GB RAM, Ubuntu 22.04, and a local NVMe drive.

Measured performance (cold start → warm-up)

• The hit rate rises with use: the engine pins the experts that are actually routed to, so it gets faster the longer it runs.

• Full hands-on video: https://youtu.be/jxML3S5C-8Y

• Additional comment (Apple M5 Max 128GB): CPU/default, 1.06 tok/s (RSS 21.8GB); after warm-up with Metal cu --ram 96, 1.11→1.83 tok/s; with --ram 110, up to about 2.06 tok/s.

• u/Sleepybear2611 (who conducted a three-day hands-on test of 754B): the rise in hit rate depends mainly on having a large amount of RAM (at RSS 113GB, about 30% of expert storage is held, and the LRU converges); on a 31GB low-memory machine, coverage can only settle at ~40–60%, because a single generation touches 38% of all experts and the working set cannot fit in the small memory; the cold-start figures match the prediction from their “bytes/token ÷ disk bandwidth” model (about 11GB/token).

A killer use case from the comments (quoted verbatim from u/trej)

"For the past month, I've been running local GLM 5.2 Q4 non-stop on my workstation rig, on repeat asking the prompt **'R… This is a necessary excerpt; read the original source for full context.

(A local, continuously running audit prompt that can be copied directly; observed result: about one round every 3.5 days, 10–15 new bugs per round, 1–2 of them hallucinations, and occasional impressive findings.)

Key points from the community discussion

• Opposition (u/Littlepharaoh): 0.5 tok/s is too slow; the same task can be completed in a few minutes with serverless Runpod or a cheap API, at a cost of a few cents. Running locally is suitable for people whose countries prohibit international payments or who do not want to hand over their codebase.

• Support (u/SV_SV_SV): a local 24/7 background audit does not give the code or data to anyone else, so the philosophical and privacy value is real.

• Ecosystem: u/misanthrophiccunt mentioned that colibrì is adding DeepSeek V4 Flash support; u/Refinery73 asked whether small or medium-sized MoE versions are available—the current engine was built specifically for the GLM series (proof of concept).

Evidence highlights and scope

• What the conclusion covers: GLM 5.2 (744B MoE) can run on a machine with no GPU using only RAM + NVMe (pure CPU), making it suitable for “slow, background, long-running” code-audit tasks; its real throughput is far below that of cloud APIs, so it is not suitable for interactive development.

• Parameter naming: The post calls it 744B (other sources, such as the GLM-5.3 release post, call it 743B; this may be a rounding difference); after int4 quantization, it is about 370GB.

• Boundary: This is an experience report from a single-user environment with 132GB RAM; hit rate and throughput can drop substantially on machines with less memory. “10–15 bugs/3.5 days” is an observation from one user repeatedly running a single prompt, not a controlled benchmark.

• Reproduction: The same order of magnitude can be reproduced with the colibrì engine, GLM-5.2 int4 weights, and the hardware described above; see the original post and video for the specific parameters.