AI Infra Deep Dive: Linear Attention from DeltaNet and GDN to KDA
A 30,000-word walkthrough of how linear attention evolved — vanilla linear attention, DeltaNet, GDN and the by-channel forget gates of KDA — retold through chunk-wise parallelism, with full derivations and runnable code.
The article traces the family tree step by step: removing the denominator and introducing convolution, DeltaNet's write gates and residual writes, GDN's added forget gates, and finally KDA's per-channel forget gates — framing each step as a hypothesis about memory rather than a bag of tricks. It then re-derives everything from the training side: the limits of the RNN paradigm, sequential and causal parallelism, how transformers parallelize attention, why chunk-wise linear attention is the practical middle ground, and the Rope-versus-NoPE decision. Implementation follows: KDA's concrete implementation, how Prefix Cache interacts with recurrent state, and how speculative decoding restores state after verification. Nineteen code blocks and twenty-two tables carry the derivations. The motivation is stated up front: no one can fully explain why reasoning emerges in large models, so this remains an experimental science — hence the insistence on deriving each step instead of accepting "it just works".
The full text of this article is currently available in Chinese only. Read the original