Alibaba’s Ali releases Qwen3.8-Flash-Next: 125-billion-parameter model with only 6 billion active

parameters per token Alibaba’s Qwen team announced Qwen3.8-Flash-Next, a multimodal open-weight model designed to lower token-level cost. The backbone contains 125 billion parameters, a 51 billion-entry N-gram embedding table, and a 4 billion-parameter multi-token prediction module, totaling 180 billion parameters on disk. In practice, only 6 billion parameters are activated for each token, thanks to a Mixture-of-Experts (MoE) layer with 512 experts that routes through 10 shared experts and one common expert. The team describes the release as a preview of the architecture that will underpin Qwen4, analogous to how Qwen3-Next preceded Qwen3.5.

Four new components drive the architecture. A hybrid attention mechanism combines Gated DeltaNet—a linear attention layer that compresses history into a fixed-size recurrent state—with Qwen Sparse Attention (QSA), which selects context at the micro-block level rather than per token. The model stacks 48 layers in a 12 × (3 × GDN + 1 × QSA) configuration, allocating a QSA budget of 512 blocks or 2 048 tokens. The residual stream expands into four parallel branches, each with component-wise read gates and scalar write gates, converging at a bottleneck of dimension 320. The new embedding table holds 20 million bigrams and trigrams in layer 2 and can be transferred to host memory with asynchronous pre-loading, currently supported only on NVIDIA devices. The training recipe replaces batch-size warmup with a revised scaling law and adds the Muon optimizer alongside AdamW for specific weight categories.

Self-reported benchmarks show strong performance on code-related tasks and agentic benchmarks but weaker results on high-level reasoning. Scores include 58.7 on DeepSWE 1.1, 62.5 on SWE-bench Pro, 81.0 on SWE-bench Multilingual, and 91.9 on LiveCodeBench v6. Agentic tasks yield 73.9 on CoWorkBench, 55.7 on JobBench, and 73.5 on Toolathlon Verified. Multimodal evaluations record 84.5 on AndroidWorld, 76.6 on LVBench, 88.5 on RealWorldQA, and 95.7 on MathVision with code interpreter. The model does not lead across all fronts: Claude Opus 4.6 (Max) reaches 40.0 on HLE versus 35.9 for Qwen, and DeepSeek-V4-Flash-0731 tops NL2Repo-Bench with 54.2 versus 48.1 for Qwen. The primary gap remains in frontier reasoning.

The checkpoint size is 172.78 GiB (≈185 GB) in FP8 and about 335.28 GiB (≈360 GB) in BF16, exceeding the capacity of a single workstation. According to vLLM configurations, TP2 is the minimum verified for FP8 on GB300, TP4 is recommended, and on an 8 × H200 node TEP8 is required because TP8 is incompatible with 128-wide quantization blocks. Sparse execution saves computation, not storage. The native context window is 262 144 tokens and can be extended to one million with YaRN. The model is served via vLLM, SGLang, TokenSpeed, Transformers Serve, and llama.cpp for GGUF builds. Fine-tuning is supported in Unsloth, Swift, and LLaMA-Factory. It already runs in “Standard” mode on QwenWork and works with Qwen Code. The “thinking” mode is enabled by default with reasoning_effort set to xhigh, medium, or low; the recommended temperature is 1.0 with top_p 0.95 in thinking mode and 0.7 with top_p 0.80 in instruct mode.

The team reports training cost at roughly one-ninth that of Qwen3.7-Plus. Kernel speedups are cited as up to 7.6× in prefill and 4.9× in decode per million tokens for QSA, while SGLang recipes and vLLM configurations claim 10.2× and 6.6× respectively, figures that the authors note should be treated as provisional pending independent measurement. They also claim an 8.6× prefill throughput relative to Qwen3.7-Plus with a 90 % reduction in prefix-cache usage.