Perplexity releases Lily, a single-model inference engine for Qwen3.6 on Apple Silicon

Perplexity published Lily yesterday, the local inference engine that powers Hybrid Compute in Perplexity Computer, and released it as open source. Lily is not a general-purpose library. It is a single-process runtime built for one model — Qwen3.6-35B-A3B — and one hardware family: Apple Silicon. A Rust layer loads the checkpoint and drives the generation loop. An OpenAI chat-completions compatible API streams tokens. Hand-written Metal kernels execute the model. Neither PyTorch nor MLX appears in the execution path; the deliberate narrowness is the performance argument.

Model architecture and three distinct load patterns

Qwen3.6-35B-A3B holds 35 billion parameters and activates roughly 3 billion per token. A router ranks 256 experts and selects eight, alongside one shared expert that sees every token. The model mixes 10 full-attention layers using Grouped-Query Attention (16 query heads, two key/value heads) with 30 Gated DeltaNet layers, a scheme that creates three distinct load patterns: uneven expert groups, attention over a growing KV cache, and fixed-size recurrence. Each pattern demands a different execution plan, and Lily implements all three in dedicated kernels.

Prefill: packed weights, routing on the GPU

The 4-bit checkpoint uses grouped affine quantization: every group of 64 weights shares a scale and bias in bfloat16, compressing 70 GB of bfloat16 weights to 19.4 GB. Metal 4 tensor operations consume bfloat16, so the weights must be reconstructed on the fly. Lily does this one tile at a time inside the batched GEMM, holding results in threadgroup memory and accumulating in FP32 so the expanded array never reaches unified memory. In Perplexity's ablation, this fusion lifted end-to-end prefill by 77.4% on a 512-token prompt. Keeping the routing histogram, prefix scan, scatter, and block mapping inside a single GPU command buffer added 89% at the same length by removing a CPU synchronization at every MoE layer. Moving from 16-row tiles to 32 with four simdgroups contributed 13.2% at 2K tokens, and a register-resident Gated DeltaNet scan added 5.6%. Roughly 90% of prefill time is spent in expert GEMMs, so long prompts run in blocked chunks to keep temporary activations from competing with weights and cache for memory.

Decode: minimizing bytes moved per token

In batch-1 decode there is almost no weight reuse, so memory bandwidth sets the ceiling. A single recorded step fired 795 kernels that produced 555 sequential stages; Lily records true dependencies in a Metal parallel pass so independent kernels overlap. The chosen token is written directly into the reserved input slot of the next step on the GPU, eliminating a CPU round trip per token, and four fused kernel chains keep intermediate values in registers. Coalesced cache reads raised key bandwidth from 33.8 to 47.9 GB/s and value bandwidth from 42.0 to 61.8 GB/s. GQA packing — four query heads sharing one threadgroup so each KV row is loaded once — improved decode by 23.8% at 32K context. A blocked attention layout at 32K and above yielded 7.7% at 32K, 27.4% at 64K, and 40.2% at 128K.

Results versus MLX-LM on M5 Max

On a Mac M5 Max with 40 cores and 128 GB of memory, running batch 1 with the same 4-bit checkpoint bytes, Perplexity measured ten context lengths from 256 to 128K tokens against MLX-LM's fastest path. Lily averaged 4,156 tokens per second in prefill versus 3,388 (1.23×) and 170.0 tokens per second in decode versus 126.4 (1.35×). At a 4K prompt with 4K context it reached 5,749.9 tokens per second in prefill and 186.6 in decode. The standalone demo is available in the pplx-garden repository; it requires a Mac with at least 24 GB of unified memory and 32 GB for optimal results, on macOS 15 or later.