Atlas · GenAI 2026
Information Theory
Information theory (entropy, perplexity, cross-entropy)
conceptPeak: 2019Information TheoryAI consensus: 2/3
Prerequisites
No prerequisites.
Recommended reference
MacKay, D. (2003) Information Theory, Inference, and Learning Algorithms — free PDF; Ch.1-6 brilliantly connect info theory to ML
Notes from AI deep research
Anthropic Opus
Cross-entropy loss = fundament treningu LLM. Perplexity = miara jakosci. KL divergence = miara odleglosci rozkladow
Google Deep Think
Miary jakości modeli językowych [G#8]
Related skills
- → is subcategory of: Probability Theory(2/3)
- → is part of: Deep Learning(1/3)
- → is part of: Large Language Models (LLM)(1/3)