文档介绍：August 2007 Trevor Hastie, Stanford Statistics 1
Least Angle Regression
Brad Efron .∗(2004)
Stanford University
∗ Trevor Hastie, Ian Johnstone, Rob Tibshirani
August 2007 Trevor Hastie, Stanford Statistics 2
August 2007 Trevor Hastie, Stanford Statistics 3
The Beginning
It all started with this picture on page 330 in our Elements of
Statistical Learning (2001). This picture links boosting to the lasso,
and is intended to explain how boosting fits models in
high-dimensional space.
Lasso Forward Stagewise
lcavol lcavol
svi svi
lweight lweight
pgg45 pgg45
lbph lbph
Coefficients
Coefficients
gleason gleason
age age
-
-
P lcp lcp
| | 0 50 100 150 200 250
t = j βj Iteration
August 2007 Trevor Hastie, Stanford Statistics 4
Adaboost Stumps for Classification
Adaboost Stump
Adaboost Stump shrink
Test Misclassification Error Test Misclassification
0 200 400 600 800 1000
Iterations
August 2007 Trevor Hastie, Stanford Statistics 5
Least Squares Boosting with Trees
Elements of Statistical Learning (chapter 10)
Response y, predictors x =(x1,x2 ...xp).
1. Start with function F (x) = 0 and residual r = y
2. Fit a CART regression tree to r giving f(x)
3. Set F (x) ← F (x)+f(x), r ← r −f(x) and repeat steps 2
and 3 many times
August 2007 Trevor Hastie, Stanford Statistics 6
Least Squares Boosting for Linear Regression
Here is a version of least squares boosting for linear regression:
(assume predictors are standardized)
(Incremental) Forward Stagewise
1. Start with r = y, β1,β2,...βp =0.
2. Find the predictor xj most correlated with r
3. Update βj ←βj + δj, where δj = · signr, xj
4. Set r ← r −δj · xj and repeat steps 2 and 3 many times
δj = r, xj gives usual forward stagewise; different from forward
stepwise
Analogous to least squares boosting, with trees=predictors
August 200