文档介绍：A Brief Introduction to:
Information Theory, Excess Entropy
and
Computational Mechanics
April 1998
(Revised October 2002)
David Feldman
College of the Atlantic
105 Eden Street
Bar Harbor, ME 04609
******@
/
i
Contents
1 Background in Information Theory 1
Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Shannon Entropy and its Many Interpretations . . . . . . . . 2
Entropy as Uncertainty . . . . . . . . . . . . . . . . . 2
Axiomatic Definition . . . . . . . . . . . . . . . . . . . 3
Shannon Entropy as Thermodynamic Entropy . . . . 4
Shannon Entropy as Average Surprise . . . . . . . . . 5
Entropy and Yes-No Questions . . . . . . . . . . . . . 5
Entropy and Coding . . . . . . . . . . . . . . . . . . . 7
Joint and Conditional Entropy . . . . . . . . . . . . . . . . . 8
Mutual Information . . . . . . . . . . . . . . . . . . . . . . . 9
Entropy of Continuous Variables . . . . . . . . . . . . . . . . 9
Continuous Entropy Discrete Entropy . . . . . . 9
←→
Careful Definition . . . . . . . . . . . . . . . . . . . . 11
2 Entropy Density and Excess Entropy 13
Entropy Density . . . . . . . . . . . . . . . . . . . . . . . . . 13
Entropy Density and Kolmogorov-plexity 16
What Entropy Density Isn’t . . . . . . . . . . . . . . . 16
Entropy Growth and Convergence . . . . . . . . . . . . . . . 17
History of Excess Entropy . . . . . . . . . . . . . . . . . . . . 20
Transient Information . . . . . . . . . . . . . . . . . . . . . . 21
putational Mechanics 23
Causal States and -machines: Preliminary Examples . . . . . 24
Example 1: A Fair Coin . . . . . . . . . . . . . . . . . 25
Example 2: Period 1 Configuration . . . . . . . . . . . 26
Example 3: Period 2 Configuration . . . . . . . . . . . 27
Summary of Examples . . . . . . . .