01 - Shannon Entropy
Definition and properties of Shannon entropy: quantitative measure of a source's uncertainty. Formula, joint and conditional entropy, chain rule.
Information Theory
Series “Information Theory”: it starts with “01 - Shannon Entropy”. Definition and properties of Shannon entropy: quantitative measure of a source's uncertainty. Formula, joint and conditional entropy, chain rule.
This series collects 6 articles (approximately 63 minutes of total reading time), designed for a level Advanced. The main topics covered are teoria informazione, entropia, shannon, teoria dell'informazione.
6Articles63 minTotal reading timeAdvancedLevel
teoria informazioneentropiashannonteoria dell'informazioneinformazione mutuadivergenza kullback-leibler
Series Articles
- 1
- 2
02 - Mutual Information and Kullback-Leibler Divergence
Mutual information I(X;Y): how much Y reduces uncertainty about X. Kullback-Leibler divergence: measure of distance between distributions.
- 3
03 - Source Coding: Huffman and Shannon's First Theorem
Prefix codes and Kraft inequality. Huffman code construction. Shannon's first theorem: average length of any code >= H(X). Optimal compression.
- 4
04 - Noisy channels and capacity: from error probability to Shannon's second theorem
The binary symmetric channel, channel capacity, and Shannon's second theorem: the foundation of reliable communication over real channels.
- 5
05 - Error-correcting codes: Hamming distance, Hamming and Singleton bounds
Hamming distance, linear block codes, Hamming and Singleton bounds for error correction: from theory to practice on disks and networks.
- 6
06 - Lossy compression and rate-distortion: Shannon's third theorem and model quantization
Shannon's rate-distortion theory: compression with distortion constraint, the third theorem, and the bridge to machine learning model quantization.
