Kimi Delta Attention
KDA replaces the growing KV cache with one fixed-size matrix S — an associative memory of size d_k×d_v. Every token edits it in three moves: forget the old contents channel-by-channel, delete whatever value the current key already points to, then write the new binding. Step through one sequence and watch the memory take shape.
An 8 by 8 heatmap shows the recurrent state matrix. Controls step through five tokens; each token applies a forget, delete, write, and read phase to the state.
The three key channels of token 3 overlap token 1's, so watch the delete step scrub token 1's stored value before token 3 writes its own — that overwrite is the delta rule earning its keep. In the real model α and β are produced per token from the input; here α is held fixed so the channel-wise decay (rows 7–8 fade fastest) stays legible.