Inside Kimi K3 Technical Report

Last Updated on July 30, 2026 by Editorial Team Author(s): Mengliu Zhao Originally published on Towards AI. Kimi-series latest model, K3, got scaled up to 2.8 trillion parameters. Impressive. Moonshot AI’s Kimi K3 technical report opens with a model that is, on paper, almost three times the size of Kimi K2–2.8T total parameters, 104B activated, a 1M-token context window, and native vision. The loss comparison shows a 2.5X scaling efficiency over Kimi K2 — just another proof that the scaling law still has its room. However, the scaling law itself is just a piece of evidence on “what works” — but the underlying question, “how things work”, is the actual key in the report. In this article, I’ll walk through four architectural ideas that make Kimi K3’s scale tractable: Core attention architectures: Linear attention with Kimi Delta Attention (KDA) and Gated MLA with NoPE Attention residuals across depth Latent MoE with quantile-based load balancing And a vision tower trained without any contrastive pre-training Then close with the infrastructure co-design that had to happen alongside all of it, continuing a trend that started with DeepSeek V3. Image source: https://tinyurl.com/mu6ezvf3. Author: Zetong Li Core attention architectures The core attention layer is made of 3 KDA layers + 1 Gated MLA layer, each followed by a stable latentMoE layer (which we’ll talk about laer). First, we’ll explain the KDA layer. Kimi Delta Attention (KDA) Kimi K3 doesn’t use classical multi-head attention as its default sequence mixer. Instead, three out of every four attention layers are Kimi Delta Attention (KDA), a linear attention mechanism first proposed in Kimi Linear (2025) and directly descended from Gated DeltaNet. The DeltaNet idea is simple: to address the underperforming issue caused by the vanilla linear attention design, a delta rule based on generalized Householder transformation was proposed to update the linear recurrence: DeltaNet update rule. Equation source: https://arxiv.org/pdf/2510.26692 Then Gated DeltaNet further added a scalar forget gate to further stabilize the learning process: Gated DeltaNet update rule. Equation source: https://arxiv.org/pdf/2510.26692 Which Kimi Delta Attention (KDA) further added a diagonalized gate to enable fine-grained control of the decay: KDA update rule. Equation source: https://arxiv.org/pdf/2510.26692 To improve computational efficiency, KDA could be further rewritten into a chunk-wise parallel format (which is a typical format for linear attention computation). To address the precision overflow issue, a negative-softplus mapping is introduced to bound the decay logits. Attention with KDA. Image source: https://arxiv.org/pdf/2607.24653 Gated MLA & No Position Encoding (NoPE) The Multi-head Latent Attention (MLA) is a mechanism DeepSeek-V2 introduced to shrink the KV cache by compressing keys and values into a shared low-rank latent before reconstructing per-head projections at attention time. While classical MLA applies RoPE to inject positional information, Kimi K3 drops this entirely — its MLA layers use NoPE (No Position Encoding). The reasoning is architectural: the KDA layers already provide position-sensitive, recency-aware mixing through their recurrent decay, so the MLA layers are freed up to do what they’re best at. It also sidesteps a very practical long-context headache: no RoPE frequency base to retune, no YaRN interpolation needed when extending context length, since there’s no rotary embedding to extrapolate in the first place. On top of this, K3 adds a full-rank, input-dependent output gate to MLA (mirroring a similar gate added to KDA), letting each token modulate which channels it reads from global attention. Multi-head latent attenion from DeepSeek paper. Image source: https://arxiv.org/html/2412.19437 Attention Residuals: Treating Depth Like a Sequence This is probably one of the most interesting concepts proposed by the technical report: similar to how transformers addressed sequential dependencies with attention, the Attention Residual attempts to formulate the residuals in attention format: Duality of depth and residual. Equation source: https://arxiv.org/pdf/2603.15031 When moving into Kimi K3 architecture, it means calculating the residual per-pair of layers: Full attention residual definition. Equation source: https://arxiv.org/pdf/2607.24653 Given the model depth < 100, the full computation cost is O(L²d), which is still affordable, and can be reduced if the residual is computed in a block-wise style. So why adding residuals in such a layer-wise style? The reason is simple — Kimi K3 scaled the number of layers up to 93, comparing Kimi K2 which only has 61 layers. The 52% incease of model layers causes a huge bottleneck on gradient backpropagation — which cannot be resolved by classical skip connection, and needs more dense representation. Full Attention residual representation. Image source: https://arxiv.org/pdf/2603.15031 Stable LatentMoE: Scaling to ~1,000 Experts Without Losing Balance Kimi K3 pushes the number of experts in its MoE architecture much further — 896 routed experts with 16 activated per token (versus K2’s 384 routed / 8 active) — and that 133% extra jump in sparsity breaks two things that used to work fine at smaller scale. The architectural fix is LatentMoE: shared experts still operate on the model’s full hidden width, but routed experts operate in a much narrower latent space, reached via a down-projection before dispatch and an up-projection after aggregation. This decouples the router’s cost from the full model width, which is what makes activating 16 of 896 experts per token affordable at all. The stability problems remain, though. Kimi K3 addresses this two approaches: i) with an RMSNorm before the up-projection + a new activation function, SiTU-GLU (Sigmoid Tanh Unit GLU), which soft-caps both branches of a SwiGLU-style gate with a scaled tanh — keeping the near-origin behaviour that makes SwiGLU work well, while bounding the output so large-magnitude coordinates can’t blow up in low precision; ii) Quantile Balancing: directly sets the expert bias to the score-quantile that would deliver that expert its target load, estimated per training step from a histogram of routing margins (to avoid gathering millions of individual values for an exact quantile). Shared and routed MoE design. Image source: https://arxiv.org/pdf/2607.24653 A Vision Tower That Doesn’t Need Contrastive Pre-training One of the more surprising empirical results in the report has nothing to do with attention or MoE. Kimi K2.5’s vision encoder, like most multimodal LLM vision towers, was initialized from a contrastively pre-trained […]