SQLite AI's 6,000-line C engine runs full Kimi K3 on 64 GB MacBook Pro

A group calling itself SQLite AI has released an inference engine named WASTE that runs the full version of Kimi K3 — a mixture-of-experts model of nearly three trillion parameters — on a laptop with 64 GB of unified memory. No BLAS, no CUDA, no Python at runtime. The engine is written in pure C, weighs in at roughly 6,000 lines, and relies on expert streaming: pulling experts from the NVMe SSD directly into compute as they are needed. The full weights, after conversion from safetensors into the engine's proprietary format, occupy 982 GiB — about 1.05 TB on disk. They do not fit in memory, not even close.

K3 is a MoE model with 92 layers and 16 experts per layer. In each single-token forward pass only about 4 % of the weights are activated, the experts relevant to that token. WASTE keeps a resident footprint of 27.28 GB in memory and fetches experts from the NVMe precisely when required. Each expert is stored in residual vector quantization with three codebook stages of 256 entries each, three bits per weight. The gate, up and down matrices of every expert sit contiguously on disk, so loading one expert equals a single sequential read. The result: 0.49–0.54 tokens per second, full weights, no distillation, no layer pruning.

The bottleneck is the disk, not the compute. Per token the engine reads 17 GB from the SSD. The MacBook's internal NVMe delivers 12.78 GB/s and the model keeps pace. The same experiment on an external USB drive at 0.94 GB/s drops generation to 13 seconds per token — territory for the very patient only. The more revealing number comes from the memory side: growing the expert cache above 46 GB not only fails to help, it destroys performance. At 52 GB speed falls by a third; at 58 GB by a factor of eight. The cause: macOS begins evicting the cache to swap, and every page fault becomes a disk access. WASTE caps itself in advance at one working set below the theoretical maximum.

Half a token per second works out to roughly 30 seconds per sentence. But models a quarter the size still run on servers with terabytes of DDR5; here nearly three trillion parameters answer without a network. For those wanting to start smaller, the same engine runs Kimi-Linear 48B from a 19 GB container at 10.7 tokens per second. K3 requires clearing a terabyte of disk space and about five hours of conversion on an M5 Pro using three parallel processes, or nearly a full day if the conversion runs in pure PyTorch. The authors, incidentally, maintain a log of falsified hypotheses documenting every measurement they took and discarded. The code is Apache 2.0 and available on GitHub.