Posts tagged with "Shannon"
2 posts found
Mar 31, 2026 information theory Shannon channel capacity attention bounds mutual information rate-distortion
The Information-Theoretic Limits of Context Windows
There are fundamental limits on how much information a fixed-width attention mechanism can extract from n tokens. Here's the math from Shannon's channel capacity to attention bounds.
Mar 31, 2026 rate-distortion context compression Shannon lossy compression information theory context compaction
The Rate-Distortion Theory of Context Compression
Context compaction is a lossy compression problem. Rate-distortion theory gives the theoretical lower bound on how much conversation history can be compressed.