Moonshot AI simply launched Kimi K3. It’s a 2.8-trillion-parameter mannequin with native imaginative and prescient and a 1-million-token context window. Moonshot calls it the world’s first open 3T-class mannequin.
What’s Kimi K3?
Kimi K3 is a sparse Combination-of-Specialists (MoE) mannequin constructed on two architectural updates. These are Kimi Delta Consideration (KDA) and Consideration Residuals (AttnRes). Each change how data flows throughout sequence size and mannequin depth. K3 targets long-horizon coding, information work, and reasoning.
Moonshot workforce states K3 is the primary open mannequin to achieve 2.8 trillion parameters. For 9 of the previous twelve months, Kimi fashions set the higher sure of open-model sizes.
Moonshot can also be direct about the place K3 sits. General efficiency nonetheless trails essentially the most highly effective proprietary fashions, Claude Fable 5 and GPT 5.6 Sol. Throughout Moonshot’s personal analysis suite, K3 persistently outperformed different examined fashions.
The Structure Beneath
Kimi Delta Consideration (KDA) is a hybrid linear consideration mechanism. Moonshot states it permits as much as 6.3x quicker decoding in million-token contexts.
AttnRes works alongside the opposite axis, which is depth. It selectively retrieves representations throughout depth fairly than accumulating them uniformly. Moonshot states AttnRes delivers roughly 25% increased coaching effectivity at below 2% further value.
Sparsity is the third lever. K3 makes use of Steady LatentMoE, successfully activating 16 of 896 specialists. At that sparsity, routing and optimization develop into first-order challenges. Quantile Balancing derives professional allocation instantly from router-score quantiles. That eliminates heuristic updates and a delicate balancing hyperparameter. Per-Head Muon extends Muon by optimizing consideration heads independently. Sigmoid Tanh Unit (SiTU) and Gated MLA enhance activation management and a spotlight selectivity respectively.
Refined coaching and information recipes accompany these structural adjustments. Collectively they yield roughly 2.5x higher total scaling effectivity than Kimi K2.
These decisions carry into serving. K3 applies quantization-aware coaching from the SFT stage onward. It makes use of MXFP4 weights with MXFP8 activations for broad {hardware} compatibility. Moonshot workforce recommends supernode configurations with 64 or extra accelerators. As a result of KDA poses new challenges for prefix caching, Moonshot contributed an implementation to vLLM.
