文档介绍：Outline
Maximum Entropy Method
Chapter 9: Methods of Entropy
Jiangsheng Yu
School of Electronics Engineering puter Science
Peking University, Beijing 100871, China
Statistical Machine Learning, 2008
Jiangsheng Yu Methods of Entropy
Outline
Maximum Entropy Method
Outline of topics
1 Maximum Entropy Method
Maximum entropy prior (MEP)
Example of maximum entropy prior
Jaynes entropy, 1968
Feature-based Model
MLE of feature-based model
Improved iterative scaling (IIS) algorithm
Jiangsheng Yu Methods of Entropy
Maximum entropy prior (MEP)
Outline
Feature-based Model
Maximum Entropy Method
Improved iterative scaling (IIS) algorithm
Maximum entropy prior (MEP)
Let π be a probability density on discrete Θ.
X
H(π) = −π(θi ) log π(θi ) (1)
Θ
Theorem (Maximum entropy prior)
Given the partial prior information about θ in the form of restrictions
π X
E [gk (θ)] = π(θi )gk (θi ) = µk (2)
i
where k = 1, 2, · · · , m. Then the MEP is
Pm
exp{ k=1 λk gk (θi )}
π¯(θi ) = P Pm (3)
i exp{ k=1 λk gk (θi )}
where λk are constants determined by (2).
Jiangsheng Yu Methods of Entropy
Maximum entropy prior (MEP)
Outline
Feature-based Model
Maximum Entropy Method
Improved iterative scaling (IIS) algorithm
Example of maximum entropy prior
Assume Θ= N and given Eπ(θ) = 5.
By (2), m = 1, g1(θ) = θ, µ1 = 5. The MEP is
eλ1θ
π¯(θ) = = (1 − eλ1 )eλ1θ
P∞λ1θ
θ=0 e
Thus, E¯π(θ) = (1 − eλ1 )/eλ1 . Let
λ1 λ1
(1 − e )/e = µ1 = 5
we have π¯(θ) = 5/6θ+1.
Jiangsheng Yu Methods of Entropy
Maximum entropy prior (MEP)
Outline
Feature-based Model
Maximum Entropy Method
Improved iterative scaling (IIS) algorithm
Jaynes entropy, 1968
Let π0(θ) be the natural invariant noninformative prior. Jeynes defined
" π(θ) # Z π(θ)
H(π) = −Eπ log = −π(θ) log dθ(4)
π0(θ) π0(θ)
Theorem (Jaynes entropy, 1968)
The MEP restricted by (2) is
Pm
π0(θ) exp{ k=1 λk gk (θ)}
π¯(θ) = R (5)
( ) {Pm ( )}
Θπ0