文档介绍：Entropy and
Information Theory
ii
Entropy and
Information Theory
Robert M. Gray
Information Systems Laboratory
Electrical Engineering Department
Stanford University
Springer-Verlag
New York
iv
This book was prepared with LATEX and reproduced by Springer-Verlag
from camera-ready copy supplied by the author.
°c 1990 by Springer Verlag
v
to Tim, Lori, Julia, Peter,
Gus, Amy Elizabeth, and Alice
and in memory of Tino
vi
Contents
Prologue xi
1 Information Sources 1
Introduction .............................. 1
Probability Spaces and Random Variables ............. 1
Random Processes and Dynamical Systems ............ 5
Distributions ............................. 6
Standard Alphabets ......................... 10
Expectation .............................. 11
Asymptotic Mean Stationarity ................... 14
Ergodic Properties .......................... 15
2 Entropy and Information 17
Introduction .............................. 17
Entropy and Entropy Rate ..................... 17
Basic Properties of Entropy ..................... 20
Entropy Rate ............................. 31
Conditional Entropy and Information . . ............. 35
Entropy Rate Revisited ....................... 41
Relative Entropy Densities ...................... 44
3 The Entropy Ergodic Theorem 47
Introduction .............................. 47
Stationary Ergodic Sources ..................... 50
Stationary Nonergodic Sources ................... 56
AMS Sources ............................. 59
The Asymptotic Equipartition Property . ............. 63
4 Information Rates I 65
Introduction .............................. 65
Stationary Codes and Approximation . . ............. 65
Information Rate of Finite Alphabet Processes .......... 73
vii
viii CONTENTS
5 Relative Entropy 77
Introduction .............................. 77
D