Meta’s Muse Glimmer 30B runs locally via GGUF on llama.cpp

The Atomic-Chat project released the first GGUF versions of Muse Glimmer 30B, Meta’s multimodal model, allowing it to be run completely offline on local hardware. Quantization was performed from the original BF16 weights, and support was merged into llama.cpp master (commit 62bf73d, PR 26841), so users must build from source or wait for a nightly binary that contains this commit.

The first stage includes a full-precision base and 8-bit versions, while the full scale with a tuned importance matrix and NVFP4 is still under development. Q8_0 runs on a single RTX 5090 and provides 35.7 tokens per second; adding the mmproj for image input pushes the workload to a second card. For normal use the recommended options are Q8_0 or AD-Q8_0, both files located in experiments/ isolate a large tensor each in BF16 and are intended only for comparison. The pre-tensor composition of each file is described in layouts/*.json, and the raw logs behind every number are published in the dedicated repository.

Running the model requires the --jinja flag; without it the chat template and tool-call parsing do not work. A basic launch example is:

```

llama-server -m Muse-Glimmer-30B-Q8_0.gguf \

--mmproj mmproj-Muse-Glimmer-30B-BF16.gguf \

--jinja -ngl 99 -c 16384 -fa on \

--temp 1.0 --top-p 0.95 --top-k 64

```

Reasoning depth is controlled from the system-prompt line with “Reasoning strength: low|medium|high|xhigh”, defaulting to high. There is no separate flag for this setting.

Speculative decoding looks good on paper but performs less impressively in practice. With DFlash in BF16 the acceleration is theoretical, but a benchmark of nine prompts (code, prose, reasoning) yielded a speed-up of only 1.9× compared with the 3.1× reported by Meta for a single RTX 5090. The gap likely stems from layer splitting across four cards, making each target forward pass relatively costly for the drafter. Acceptance rates vary by task: code reaches 0.37, while free-form prose falls to 0.12.

Atomic’s BF16 GGUF is bit-identical to the original weights, verified by UUID and all 731 tensor hashes. KL-divergence metrics against this reference show a neutral-text perplexity (30 languages, 97 windows of 4096) of 5.4614 and a dialogue-with-tool-calls perplexity (85 windows) of 2.4462. Lower KLD and higher top-1 indicate that the quantization preserved more information.

The file imatrix/muse-glimmer.imatrix.gguf was released so anyone can reproduce the quantizers or build their own with the same calibration. The matrix records for each weight matrix the mean-squared activations per input channel; the quantizer uses this to weight error reconstruction in block-scale searches, penalizing errors in channels with large activations more heavily. Two flags are off by default and both are critical: without --parse-special tokens such as <|start|> are expanded as literal punctuation, so 40 % of the corpus was tuned on sequences the model does not generate; without --process-output the output.weight, 1.34 billion parameters sitting directly under the softcap of the logits, does not receive importance data.