Mar 31, 2026 information theoryShannonchannel capacityThe Information-Theoretic Limits of Context WindowsThere are fundamental limits on how much information a fixed-width attention mechanism can extract from n tokens. Here's the math from Shannon's channel capacity to attention bounds.
Mar 31, 2026 rate-distortioncontext compressionShannonThe Rate-Distortion Theory of Context CompressionContext compaction is a lossy compression problem. Rate-distortion theory gives the theoretical lower bound on how much conversation history can be compressed.
Mar 31, 2026 softmaxattention dilutioncontext rotSoftmax Attention and the Dilution Problem: The Math Behind Context RotAs context grows, softmax normalizes attention weights so each relevant token gets less attention. This mathematical property is why AI accuracy drops with length.
